Proximal diffusion models: a diffusion prior that is queried through its proximal operator instead of its score. Implemented here are the interface, an exact oracle, ProxDM’s sampler, proximal inference for the implicit learner and its adjoint, ProxDM as a factor’s inversion, and the training loss.
Sources: Fang, Díaz, Buchanan & Sulam, Beyond Scores: Proximal Diffusion Models, arXiv:2507.08956 (Algorithm 1, Eq. 9); Zhu et al., Denoising Diffusion Models for Plug-and-Play Image Restoration, CVPR 2023 (DiffPIR, the half-quadratic splitting); code:
proxdm.jlBibliography: Fang et al. 2025 · Zhu et al. 2023
Theory: ProxDM and Proximal Alternatives · Deterministic Relaxation · Inference Signatures
Notation. As everywhere in this vault, is the joint state and the prior is on . The paper writes for the same thing.
1. The object
The optimality condition is a gradient step evaluated at the new point: a backward (implicit) Euler step, where a score is a forward one. That is the whole idea of ProxDM: discretise the reverse SDE backwards, and the network you need is a prox, not a score.
| type | what it is |
|---|---|
AbstractProximalPredictor | anything with proximal(p, v, t, λ, ps, st) -> (u, st) |
ProxNetwork(model, schedule; input, ad) | a Lux model in ProxDM’s parametrisation ; conditioned on two scalars |
MixtureProx(::GaussianMixtureEps) | the exact prox of a Gaussian mixture, by Newton with backtracking; the oracle a perfectly trained ProxNetwork would match |
MixtureProx uses and its Hessian directly from the mixture, not
, so it is valid at , where the sampler’s last step needs it.
2. The sampler (Algorithm 1)
On a uniform grid with :
-
PDA (
hybrid = false): , which needs ;proxdm_samplethrows otherwise. -
PDA-hybrid (
scheme = :hybrid, orhybrid = true): : the linear drift explicit, the score implicit; no step-size limit. -
Trapezoidal (
scheme = :trapezoidal): the drift averaged over both ends of the step (Crank–Nicolson),which is again one prox: with and ; it needs . The explicit score is free: the previous step returned , and the prox’s optimality condition gives . The first step is a hybrid step, which supplies the first score.
The last step evaluates the prox at , so the output is already denoised; there is no separate final denoising step.
Validation (one Gaussian, mean , std , exact prox, 1,500 samples):
| steps | 20 | 40 | 100 | 400 |
|---|---|---|---|---|
| sample std (true 0.30) | 0.248 | 0.268 | 0.286 | 0.293 |
The mean is right at every step count; the spread converges at first order, as a backward Euler scheme should. The trapezoidal scheme is second order. Its error in the spread, against PDA and the hybrid scheme (same Gaussian, 20,000 samples, common random numbers; the Monte Carlo floor is about 0.001):
| steps | 10 | 20 | 40 | 80 | 160 |
|---|---|---|---|---|---|
| PDA | −0.097 | −0.054 | −0.028 | −0.015 | −0.007 |
| hybrid | −0.099 | −0.056 | −0.030 | −0.016 | −0.007 |
| trapezoidal | −0.023 | −0.007 | −0.001 | (floor) | (floor) |
Halving the step halves the first-order error and divides the trapezoidal one by 3 to 6 before it reaches the floor. For a linear drift with constant coefficients and additive noise the trapezoidal step keeps the stationary variance exactly (the factors cancel in it), which is why the spread, the hard part for backward Euler, is right.
On the circle mixture (radial width 0.050, 3,000 samples), the radius’s spread:
| steps | 20 | 50 | 100 |
|---|---|---|---|
| PDA | 0.015 | 0.029 | 0.038 |
| hybrid | 0.016 | 0.030 | 0.038 |
| trapezoidal | 0.027 | 0.044 | 0.048 |
All three put the mean radius at 1.001 to 1.002. Fifty trapezoidal steps beat a hundred PDA steps; the remaining error sits at small , where the ring’s fine structure is and a uniform grid in is coarse (the clamped sampler of sampler uses a log-SNR grid for that reason).
The free score holds only as far as the prox is exact. A learned ProxNetwork is an
approximate prox; is then its implied score, consistent with its own
implicit steps but carrying its training error into the explicit half too.
3. Proximal inference for the implicit learner
prox_infer(p, z₀, ρ, t, ps, st; λ) is the implicit learner’s query with a proximal prior.
It splits the relaxed problem
and alternates the two proximal maps (half-quadratic splitting, as in DiffPIR):
Hard clamps are exact; the free coordinates () take . As
the coupling becomes and the problem becomes the implicit learner’s
(Deterministic Relaxation). It returns an ImplicitSolution with z = w (which satisfies
the clamps) and residual .
On the circle mixture, the query ”, which ?” from returns , symmetric to the last digit, in 20 iterations.
4. Training: proximal matching
proximal_matching_loss is one sample of the paper’s Eq. 9:
The squared loss would train the MMSE denoiser ; this bounded loss, as
, trains the MAP denoiser, which is the prox. Shrink over training.
The loss is a plain function of ps, so any Lux training loop and any AD backend trains it.
5. The adjoint of proximal inference
prox_infer returns the fixed point of the splitting. With it
satisfies, on the coordinates that are not hard-clamped (),
a root of , so the implicit
function theorem applies as it does to implicit_infer (Backpropagation through Implicit Inference).
With the prox’s input Jacobian at and , one linear solve
gives every cotangent:
| w.r.t. | cotangent |
|---|---|
| parameters of the prox | |
| hard-clamped inputs | |
| soft and free targets | |
| precisions |
prox_pullback(p, sol, z₀, ρ, t, z̄, ps, st; λ, linsolve) computes them; it is also the
ChainRulesCore.rrule of prox_infer, so Zygote differentiates through proximal inference
without unrolling it. The matrix is , not a Jacobian of a
field: the prox is already a solved problem, and its derivative is what the adjoint needs.
Only vector-Jacobian products of the prox appear, so linsolve = KrylovSolve() runs GMRES on
them and never forms .
The prox’s derivatives come from three places:
| prox | input VJP / Jacobian | parameter VJP |
|---|---|---|
MixtureProx | exact: , the implicit function theorem on the prox’s own optimality condition | exact: , valid at |
ProxNetwork(…; ad) | through DifferentiationInterface | through DifferentiationInterface |
anything else, or no ad | central differences of proximal | ArgumentError |
Validation (RING, , ). Against central differences of the
whole solve, for a hard clamp, a soft target, a soft clamp with a free coordinate, and all
coordinates soft, every cotangent (clamps, targets, precisions, mixture means) agrees to
between 1e-8 and 1e-5 relative, limited by the finite differences, not the adjoint; the test
suite asserts 1e-4 relative against a cheaper reference (below). GMRES and the dense solve agree to 1e-12; Zygote through the rrule equals
the pullback exactly. A ProxNetwork (a two-layer MLP) with AutoZygote(): input VJP and
Jacobian equal finite differences to 1e-8, a weight’s cotangent through the whole solve to
3e-6 relative.
Finite differences need care here. The splitting contracts slowly when the answer can slide along the relation (a soft clamp, or an untrained network: 350 iterations), and then its stopping rule leaves an error of about , with the contraction factor. A difference step of 1e-6 divides that by 1e-6 and disagreed with the adjoint by 1e-3; steps of 1e-4 to 1e-3 agree. Even then the reference was fragile: the same check passed alone and failed inside the full suite at 1.5e-4, so the tests run the reference solves for a fixed 1000 sweeps instead of to a tolerance. The adjoint does not have this problem: it solves the linear system to its own tolerance.
6. As a factor’s inversion
DiffusionFactor(blocks, pred; prox = ProxDM(prox; t, λ, tol, maxiters, linsolve)) makes
ProxDM usable in a factor graph, alongside REDDiff and ImplicitProx. invert and
factor_message run prox_infer on the factor’s state space with the polarity’s precisions,
warm-started at the incoming points (the same problem implicit_solution builds for
ImplicitProx), and return a DiracBelief on the target. implicit_solution returns the
report and implicit_factor_pullback the backward pass through prox_pullback.
Two parameter trees. It is the first factor in the project with two models: the
NoisePredictor (the prior; the free energy’s score term needs it) and the prox (the
inversion). Its parameters and states are (predictor = …, prox = …), set up by
LuxCore.setup(rng, f); a MixtureProx gets (μ = …,). This is the structure
ProxDM and Proximal Alternatives §4 anticipated: the inversion is a separate learned object,
not a function of the prior’s parameters, so the factor cannot be one Lux layer over one tree.
The factor-level pullback returns ps = (predictor = 0, prox = ps̄), and the tests check that
moving the prior’s means leaves the inversion unchanged while changing the score term.
The score summand uses fixed nodes (nodes, by default 8 from Xoshiro(0)) at weight
, since ProxDM’s own λ is the splitting’s coupling and has no score meaning.
Validation (RING, MixtureProx): the factor’s answer equals bare prox_infer exactly;
the two branches from opposite warm starts are symmetric to 1e-8; the pullback’s input and
precision cotangents match finite differences of invert; with a ProxNetwork, a weight’s
cotangent in the prox tree matches finite differences of the factor’s answer to 1e-3.
A dispatch trap. The methods for a ProxDM factor dispatch on
DiffusionFactor{N,D,P,<:ProxDM} where {N, D<:Tuple, P<:NoisePredictor}. Written with
unbounded D and P the signature is not a subtype of the plain DiffusionFactor signature (the
struct’s bounds are part of it), so Julia never prefers it and silently calls the generic method.
7. Implementation difficulties and open ends
- Non-convex priors make the prox multivalued. For a mixture, is not convex,
and can have several minimisers.
MixtureProxuses Newton where is positive definite and a gradient step otherwise, and returns the local minimiser reached from . A learnedProxNetworkmakes the same choice implicitly, wherever training put it. - The PDA step-size limit. fails for coarse grids near under the default VP schedule. The hybrid scheme removes it; PDA throws with a message saying so instead of returning garbage.
- Backpropagating at a different noise level gives a surrogate, not a gradient (§8). In the one experiment so far it trains no better than the exact adjoint.
- No derivative in
torλ. Both are treated as hyperparameters. stableis not assessed by the splitting; it reportstruewhen converged. The adjoint’s matrix is nonsingular wherever the splitting contracts, which is when it converges at all.- A log-SNR time grid for
proxdm_samplewould move the steps to small ; with it the first steps near get larger, which PDA’s and the trapezoidal would then hit (the hybrid first step has no limit). - Point messages only. A Laplace Gaussian for proximal inference would need the Hessian of the relaxed problem, which the prox gives only through its Jacobian ( at the prox’s output); not built.
- The two trees are separate even when they could be shared. For
MixtureProxof the factor’s own mixture, both trees hold the same means; training one does not move the other.
8. Experiment: backpropagating at a higher noise level
Backpropagation through Implicit Inference §9 asks whether the adjoint should be taken at
a noise level above the forward’s : the gradient of a more smoothed problem, biased
but better conditioned, with graduated smoothing as the hoped-for benefit. prox_pullback
takes the noise level as its own argument, so the surrogate is one call with t = t_b. Away
from the fixed-point equation no longer holds at , so the result is not the gradient of
anything.
Setup (examples/proxdm_backprop_noise.jl). The task of §7 there: 64
mixture components on the unit circle (width 0.05), means trained by Adam so that proximal
inference "" ( clamped) reproduces on 9 training inputs, with
gradients from the adjoint only. MixtureProx, (), ,
150 epochs; , i.e. from 0.027 to 0.32.
The surrogate’s direction. Cosine to the exact gradient, and its norm relative to the exact one, on the 9 training points:
| 0.01 | 0.02 | 0.05 | 0.1 | |
|---|---|---|---|---|
| at the start | 0.98 (0.77) | 0.89 (0.53) | 0.39 (0.30) | 0.28 (0.26) |
| after exact training | 0.98 (0.82) | 0.80 (0.66) | 0.47 (0.79) | 0.34 (0.43) |
Up to it points the same way and is shorter (smoother); beyond, the direction decorrelates.
Training (learning rate 0.01, five training sets: the grid and four random draws of 9 inputs; median and range):
| final loss | loss spikes (+10%) | test RMSE, warm start | test RMSE, cold start | |
|---|---|---|---|---|
| 0.005 (exact) | 1.3e-4 [4e-7, 6e-4] | 3 [0, 11] | 0.14 [0.08, 0.30] | 0.30 [0.28, 0.34] |
| 0.01 | 7.5e-5 [5e-6, 3e-4] | 5 [1, 13] | 0.14 [0.07, 0.28] | 0.29 [0.26, 0.30] |
| 0.02 | 1.1e-4 [3e-7, 3e-4] | 2 [0, 8] | 0.14 [0.09, 0.29] | 0.23 [0.15, 0.33] |
| 0.05 | 2.6e-4 [2e-5, 2e-3] | 5 [0, 8] | 0.12 [0.05, 0.31] | 0.28 [0.05, 0.32] |
| 0.1 | 6.6e-4 [3e-4, 1.7e-2] | 2 [0, 3] | 0.16 [0.12, 0.25] | 0.22 [0.10, 0.24] |
At learning rate 0.03 (grid only) the exact adjoint reaches loss 1e-3 fastest (36 epochs, against 43–53 for up to 0.05) and never does.
Reading. The hypothesis is not supported here. Surrogates up to train about as well as the exact gradient, none consistently better, and is too biased: it stalls in one of five runs. On the grid training set alone, looked best (test RMSE 0.05 against 0.08, cold start 0.05 against 0.31) and had no loss spikes against eleven; neither survived the other four sets, which is why the table reports five.
Why this is not the last word. The motivation was conditioning, and this problem has none to fix: one free coordinate, an exact prox, an adjoint solve that is a scalar division. The cases where a high- adjoint could pay off are a learned prox (whose Jacobian at small carries its errors), many free coordinates, and a forward at very small , which is where the stiffness argument applies. The 9-point task also overfits (test RMSE 0.14 at training loss 1e-4), so generalisation differences of this size are noise.
Related: implicit, reddiff, backends, analytic, ProxDM and Proximal Alternatives, Inference Signatures, Deterministic Relaxation
LP-018 stationary mixture maxima (2026-10-29)
A small optimality residual is insufficient in a nonconvex mixture. Before accepting it,
MixtureProx checks curvature; a negative eigenvalue supplies an escape direction with
backtracking on the same objective. Iteration or line-search exhaustion throws rather
than claiming a local minimiser. Global optimality remains outside the contract.
Near a minimum, rounding can hide a Newton step’s objective decrease before the residual
meets tolerance. A step within objective roundoff is accepted only if its recomputed
residual decreases; this retains the existing sampler and tight-tolerance adjoint cases.
Evidence: test/lp018.jl, “LP-018.6 symmetric mixture prox escapes a maximum”, against
the independent two-normal density and its local curvature; the existing “ProxDM: the
exact prox, the sampler, proximal inference, the loss” tests cover the previous cases.