implementation

Proximal diffusion models: a diffusion prior that is queried through its proximal operator instead of its score. Implemented here are the interface, an exact oracle, ProxDM’s sampler, proximal inference for the implicit learner and its adjoint, ProxDM as a factor’s inversion, and the training loss.

Sources: Fang, Díaz, Buchanan & Sulam, Beyond Scores: Proximal Diffusion Models, arXiv:2507.08956 (Algorithm 1, Eq. 9); Zhu et al., Denoising Diffusion Models for Plug-and-Play Image Restoration, CVPR 2023 (DiffPIR, the half-quadratic splitting); code: proxdm.jl

Bibliography: Fang et al. 2025 · Zhu et al. 2023

Theory: ProxDM and Proximal Alternatives · Deterministic Relaxation · Inference Signatures

Notation. As everywhere in this vault, is the joint state and the prior is on . The paper writes for the same thing.

1. The object

The optimality condition is a gradient step evaluated at the new point: a backward (implicit) Euler step, where a score is a forward one. That is the whole idea of ProxDM: discretise the reverse SDE backwards, and the network you need is a prox, not a score.

typewhat it is
AbstractProximalPredictoranything with proximal(p, v, t, λ, ps, st) -> (u, st)
ProxNetwork(model, schedule; input, ad)a Lux model in ProxDM’s parametrisation ; conditioned on two scalars
MixtureProx(::GaussianMixtureEps)the exact prox of a Gaussian mixture, by Newton with backtracking; the oracle a perfectly trained ProxNetwork would match

MixtureProx uses and its Hessian directly from the mixture, not , so it is valid at , where the sampler’s last step needs it.

2. The sampler (Algorithm 1)

On a uniform grid with :

  • PDA (hybrid = false): , which needs ; proxdm_sample throws otherwise.

  • PDA-hybrid (scheme = :hybrid, or hybrid = true): : the linear drift explicit, the score implicit; no step-size limit.

  • Trapezoidal (scheme = :trapezoidal): the drift averaged over both ends of the step (Crank–Nicolson),

    which is again one prox: with and ; it needs . The explicit score is free: the previous step returned , and the prox’s optimality condition gives . The first step is a hybrid step, which supplies the first score.

The last step evaluates the prox at , so the output is already denoised; there is no separate final denoising step.

Validation (one Gaussian, mean , std , exact prox, 1,500 samples):

steps2040100400
sample std (true 0.30)0.2480.2680.2860.293

The mean is right at every step count; the spread converges at first order, as a backward Euler scheme should. The trapezoidal scheme is second order. Its error in the spread, against PDA and the hybrid scheme (same Gaussian, 20,000 samples, common random numbers; the Monte Carlo floor is about 0.001):

steps10204080160
PDA−0.097−0.054−0.028−0.015−0.007
hybrid−0.099−0.056−0.030−0.016−0.007
trapezoidal−0.023−0.007−0.001(floor)(floor)

Halving the step halves the first-order error and divides the trapezoidal one by 3 to 6 before it reaches the floor. For a linear drift with constant coefficients and additive noise the trapezoidal step keeps the stationary variance exactly (the factors cancel in it), which is why the spread, the hard part for backward Euler, is right.

On the circle mixture (radial width 0.050, 3,000 samples), the radius’s spread:

steps2050100
PDA0.0150.0290.038
hybrid0.0160.0300.038
trapezoidal0.0270.0440.048

All three put the mean radius at 1.001 to 1.002. Fifty trapezoidal steps beat a hundred PDA steps; the remaining error sits at small , where the ring’s fine structure is and a uniform grid in is coarse (the clamped sampler of sampler uses a log-SNR grid for that reason).

The free score holds only as far as the prox is exact. A learned ProxNetwork is an approximate prox; is then its implied score, consistent with its own implicit steps but carrying its training error into the explicit half too.

3. Proximal inference for the implicit learner

prox_infer(p, z₀, ρ, t, ps, st; λ) is the implicit learner’s query with a proximal prior. It splits the relaxed problem

and alternates the two proximal maps (half-quadratic splitting, as in DiffPIR):

Hard clamps are exact; the free coordinates () take . As the coupling becomes and the problem becomes the implicit learner’s (Deterministic Relaxation). It returns an ImplicitSolution with z = w (which satisfies the clamps) and residual .

On the circle mixture, the query ”, which ?” from returns , symmetric to the last digit, in 20 iterations.

4. Training: proximal matching

proximal_matching_loss is one sample of the paper’s Eq. 9:

The squared loss would train the MMSE denoiser ; this bounded loss, as , trains the MAP denoiser, which is the prox. Shrink over training. The loss is a plain function of ps, so any Lux training loop and any AD backend trains it.

5. The adjoint of proximal inference

prox_infer returns the fixed point of the splitting. With it satisfies, on the coordinates that are not hard-clamped (),

a root of , so the implicit function theorem applies as it does to implicit_infer (Backpropagation through Implicit Inference). With the prox’s input Jacobian at and , one linear solve

gives every cotangent:

w.r.t.cotangent
parameters of the prox
hard-clamped inputs
soft and free targets
precisions

prox_pullback(p, sol, z₀, ρ, t, z̄, ps, st; λ, linsolve) computes them; it is also the ChainRulesCore.rrule of prox_infer, so Zygote differentiates through proximal inference without unrolling it. The matrix is , not a Jacobian of a field: the prox is already a solved problem, and its derivative is what the adjoint needs. Only vector-Jacobian products of the prox appear, so linsolve = KrylovSolve() runs GMRES on them and never forms .

The prox’s derivatives come from three places:

proxinput VJP / Jacobianparameter VJP
MixtureProxexact: , the implicit function theorem on the prox’s own optimality conditionexact: , valid at
ProxNetwork(…; ad) through DifferentiationInterface through DifferentiationInterface
anything else, or no adcentral differences of proximalArgumentError

Validation (RING, , ). Against central differences of the whole solve, for a hard clamp, a soft target, a soft clamp with a free coordinate, and all coordinates soft, every cotangent (clamps, targets, precisions, mixture means) agrees to between 1e-8 and 1e-5 relative, limited by the finite differences, not the adjoint; the test suite asserts 1e-4 relative against a cheaper reference (below). GMRES and the dense solve agree to 1e-12; Zygote through the rrule equals the pullback exactly. A ProxNetwork (a two-layer MLP) with AutoZygote(): input VJP and Jacobian equal finite differences to 1e-8, a weight’s cotangent through the whole solve to 3e-6 relative.

Finite differences need care here. The splitting contracts slowly when the answer can slide along the relation (a soft clamp, or an untrained network: 350 iterations), and then its stopping rule leaves an error of about , with the contraction factor. A difference step of 1e-6 divides that by 1e-6 and disagreed with the adjoint by 1e-3; steps of 1e-4 to 1e-3 agree. Even then the reference was fragile: the same check passed alone and failed inside the full suite at 1.5e-4, so the tests run the reference solves for a fixed 1000 sweeps instead of to a tolerance. The adjoint does not have this problem: it solves the linear system to its own tolerance.

6. As a factor’s inversion

DiffusionFactor(blocks, pred; prox = ProxDM(prox; t, λ, tol, maxiters, linsolve)) makes ProxDM usable in a factor graph, alongside REDDiff and ImplicitProx. invert and factor_message run prox_infer on the factor’s state space with the polarity’s precisions, warm-started at the incoming points (the same problem implicit_solution builds for ImplicitProx), and return a DiracBelief on the target. implicit_solution returns the report and implicit_factor_pullback the backward pass through prox_pullback.

Two parameter trees. It is the first factor in the project with two models: the NoisePredictor (the prior; the free energy’s score term needs it) and the prox (the inversion). Its parameters and states are (predictor = …, prox = …), set up by LuxCore.setup(rng, f); a MixtureProx gets (μ = …,). This is the structure ProxDM and Proximal Alternatives §4 anticipated: the inversion is a separate learned object, not a function of the prior’s parameters, so the factor cannot be one Lux layer over one tree. The factor-level pullback returns ps = (predictor = 0, prox = ps̄), and the tests check that moving the prior’s means leaves the inversion unchanged while changing the score term.

The score summand uses fixed nodes (nodes, by default 8 from Xoshiro(0)) at weight , since ProxDM’s own λ is the splitting’s coupling and has no score meaning.

Validation (RING, MixtureProx): the factor’s answer equals bare prox_infer exactly; the two branches from opposite warm starts are symmetric to 1e-8; the pullback’s input and precision cotangents match finite differences of invert; with a ProxNetwork, a weight’s cotangent in the prox tree matches finite differences of the factor’s answer to 1e-3.

A dispatch trap. The methods for a ProxDM factor dispatch on DiffusionFactor{N,D,P,<:ProxDM} where {N, D<:Tuple, P<:NoisePredictor}. Written with unbounded D and P the signature is not a subtype of the plain DiffusionFactor signature (the struct’s bounds are part of it), so Julia never prefers it and silently calls the generic method.

7. Implementation difficulties and open ends

  • Non-convex priors make the prox multivalued. For a mixture, is not convex, and can have several minimisers. MixtureProx uses Newton where is positive definite and a gradient step otherwise, and returns the local minimiser reached from . A learned ProxNetwork makes the same choice implicitly, wherever training put it.
  • The PDA step-size limit. fails for coarse grids near under the default VP schedule. The hybrid scheme removes it; PDA throws with a message saying so instead of returning garbage.
  • Backpropagating at a different noise level gives a surrogate, not a gradient (§8). In the one experiment so far it trains no better than the exact adjoint.
  • No derivative in t or λ. Both are treated as hyperparameters.
  • stable is not assessed by the splitting; it reports true when converged. The adjoint’s matrix is nonsingular wherever the splitting contracts, which is when it converges at all.
  • A log-SNR time grid for proxdm_sample would move the steps to small ; with it the first steps near get larger, which PDA’s and the trapezoidal would then hit (the hybrid first step has no limit).
  • Point messages only. A Laplace Gaussian for proximal inference would need the Hessian of the relaxed problem, which the prox gives only through its Jacobian ( at the prox’s output); not built.
  • The two trees are separate even when they could be shared. For MixtureProx of the factor’s own mixture, both trees hold the same means; training one does not move the other.

8. Experiment: backpropagating at a higher noise level

Backpropagation through Implicit Inference §9 asks whether the adjoint should be taken at a noise level above the forward’s : the gradient of a more smoothed problem, biased but better conditioned, with graduated smoothing as the hoped-for benefit. prox_pullback takes the noise level as its own argument, so the surrogate is one call with t = t_b. Away from the fixed-point equation no longer holds at , so the result is not the gradient of anything.

Setup (examples/proxdm_backprop_noise.jl). The task of §7 there: 64 mixture components on the unit circle (width 0.05), means trained by Adam so that proximal inference "" ( clamped) reproduces on 9 training inputs, with gradients from the adjoint only. MixtureProx, (), , 150 epochs; , i.e. from 0.027 to 0.32.

The surrogate’s direction. Cosine to the exact gradient, and its norm relative to the exact one, on the 9 training points:

0.010.020.050.1
at the start0.98 (0.77)0.89 (0.53)0.39 (0.30)0.28 (0.26)
after exact training0.98 (0.82)0.80 (0.66)0.47 (0.79)0.34 (0.43)

Up to it points the same way and is shorter (smoother); beyond, the direction decorrelates.

Training (learning rate 0.01, five training sets: the grid and four random draws of 9 inputs; median and range):

final lossloss spikes (+10%)test RMSE, warm starttest RMSE, cold start
0.005 (exact)1.3e-4 [4e-7, 6e-4]3 [0, 11]0.14 [0.08, 0.30]0.30 [0.28, 0.34]
0.017.5e-5 [5e-6, 3e-4]5 [1, 13]0.14 [0.07, 0.28]0.29 [0.26, 0.30]
0.021.1e-4 [3e-7, 3e-4]2 [0, 8]0.14 [0.09, 0.29]0.23 [0.15, 0.33]
0.052.6e-4 [2e-5, 2e-3]5 [0, 8]0.12 [0.05, 0.31]0.28 [0.05, 0.32]
0.16.6e-4 [3e-4, 1.7e-2]2 [0, 3]0.16 [0.12, 0.25]0.22 [0.10, 0.24]

At learning rate 0.03 (grid only) the exact adjoint reaches loss 1e-3 fastest (36 epochs, against 43–53 for up to 0.05) and never does.

Reading. The hypothesis is not supported here. Surrogates up to train about as well as the exact gradient, none consistently better, and is too biased: it stalls in one of five runs. On the grid training set alone, looked best (test RMSE 0.05 against 0.08, cold start 0.05 against 0.31) and had no loss spikes against eleven; neither survived the other four sets, which is why the table reports five.

Why this is not the last word. The motivation was conditioning, and this problem has none to fix: one free coordinate, an exact prox, an adjoint solve that is a scalar division. The cases where a high- adjoint could pay off are a learned prox (whose Jacobian at small carries its errors), many free coordinates, and a forward at very small , which is where the stiffness argument applies. The 9-point task also overfits (test RMSE 0.14 at training loss 1e-4), so generalisation differences of this size are noise.

Related: implicit, reddiff, backends, analytic, ProxDM and Proximal Alternatives, Inference Signatures, Deterministic Relaxation

LP-018 stationary mixture maxima (2026-10-29)

A small optimality residual is insufficient in a nonconvex mixture. Before accepting it, MixtureProx checks curvature; a negative eigenvalue supplies an escape direction with backtracking on the same objective. Iteration or line-search exhaustion throws rather than claiming a local minimiser. Global optimality remains outside the contract. Near a minimum, rounding can hide a Newton step’s objective decrease before the residual meets tolerance. A step within objective roundoff is accepted only if its recomputed residual decreases; this retains the existing sampler and tight-tolerance adjoint cases. Evidence: test/lp018.jl, “LP-018.6 symmetric mixture prox escapes a maximum”, against the independent two-normal density and its local curvature; the existing “ProxDM: the exact prox, the sampler, proximal inference, the loss” tests cover the previous cases.