The Lagrangian of the original sketch, carried through: state equation, adjoint equation and gradient for an implicit diffusion learner. Also covered: the hard/soft clamp split, what the adjoint costs, why no noise has to be stored, and what to do when inference has not reduced the energy. Every gradient below is checked against finite differences of the full solve in
lib/VariationalDiffusion.jl/test/runtests.jl.
Sources: original to this vault (design and analysis); the adjoint method as in Backpropagation by the Implicit Function Theorem and DEQ as a Relation §4; Bai, Kolter & Koltun, Deep Equilibrium Models, NeurIPS 2019; Fung et al., JFB: Jacobian-Free Backpropagation for Implicit Networks, AAAI 2022; Mardani et al. arXiv:2305.04391; for §9: Pedregosa, ICML 2016; Grazzi et al., ICML 2020; Geng et al., NeurIPS 2021; Poole et al., ICLR 2023; Clark et al., ICLR 2024; Prabhudesai et al. 2023; Graikos et al., NeurIPS 2022; Mobahi & Fisher, EMMCVPR 2015; code:
implicit.jl(implicit_pullback),analytic.jl(epsilon_jacobian,epsilon_vjp_params)Bibliography: Bai et al. 2019 · Wu Fung et al. 2022 · Mardani et al. 2024 · Pedregosa 2016 · Grazzi et al. 2020 · Geng et al. 2021 · Poole et al. 2023 · Clark et al. 2024 · Prabhudesai et al. 2023 · Graikos et al. 2022 · Mobahi & Fisher 2015
Theory (CT-ML wiki): Lens · Reverse Derivative Category · Statistical Game · Compiler Correctness
1. Setup
From Implicit Diffusion Learners: free coordinates (finite precision), clamped coordinates (precision ), and the residual on
with the noise nodes fixed (sample-average approximation; why this matters is §5). Inference returns with and . Its Jacobian on the free block is
A downstream loss (e.g. ) has cotangent .
2. The Lagrangian
State equation:
This is the inference problem: solve it first, with any of the point signatures of Inference Signatures.
Adjoint equation:
One linear solve with the transpose of the free-block Jacobian. For an exact score is symmetric (Implicit Diffusion Learners §3), so the transpose is irrelevant; for a learned network it is not symmetric, and the transpose matters. Solve it densely for small (form from VJPs) or matrix-free with GMRES for large ; each GMRES iteration costs one VJP of per node.
Gradient: read everything off
By the implicit function theorem, along every input direction. Embedding in with zeros on :
| w.r.t. | gradient | needs |
|---|---|---|
| network parameters | one parameter-VJP per node | |
| hard-clamped inputs | the off-diagonal Jacobian block | |
| soft targets / anchors | nothing extra | |
| precisions | nothing extra |
The last row means the polarity weights are learnable. Soft-clamp precisions can be trained by the same adjoint, so “how much to trust an input” becomes a parameter rather than a hyperparameter.
implicit_pullback(m, sol, z₀, ρ, z̄, ps, st) returns (z₀ = …, ρ = …, ps = …). In the
backpropagation signature of the sketch it is
and the incoming energy does not need its own cotangent (§6).
3. Why this is the right derivative
The adjoint gives the exact derivative of the map defined by the deterministic residual, wherever is invertible. Checked against central finite differences of the full solve (tolerance ), on a circle relation with exact score:
| derivative | adjoint | finite difference |
|---|---|---|
| (hard input) | −0.06184743 | −0.06184742 |
| (a mixture mean, i.e. ) | +0.02653255 | +0.02653255 |
| (soft input) | +0.02502245 | +0.02502245 |
| +0.01145560 | +0.01145560 | |
| −0.00007650 | −0.00007650 |
For a Gaussian prior everything is linear: and , matched to .
4. The cost
Per backward pass: one Jacobian evaluation (or one GMRES solve), and one parameter-VJP of per noise node. No unrolling, so memory is independent of the number of inference iterations, the defining advantage of implicit differentiation. With nodes this is network VJPs, the same as one step of RED-Diff with samples, done once rather than once per iteration.
5. Does the noise from inference have to be stored?
No, and the reason is structural. There are three cases:
- Fixed nodes (what the package does). The noise is part of the model, a fixed quadrature of the expectation. The backward pass re-evaluates the Jacobians at with the same nodes; nothing from the forward trajectory is needed except .
- Fresh noise in inference (RED-Diff as published). The forward pass is a stochastic approximation whose limit is a root of the expected field . The IFT applies to , at the limit point. The trajectory’s noise is irrelevant. The backward pass needs unbiased estimates of and , which fresh noise provides.
- The catch in case 2. involves an inverse of an expectation, and . Plugging a noisy Jacobian into the solve gives a biased gradient. Fix: use many nodes in the backward pass, or the same fixed node set for the Jacobian and the residual. Fixing the nodes is the simplest way to make the whole thing exact.
So save , not the noise. If the noise is fixed it is part of ‘s computational graph anyway.
6. When the energy is not reduced
The IFT is a statement about a root. At a point that is not a root it computes the derivative
of nothing in particular. implicit_pullback therefore refuses non-converged solutions. The
options, from most to least principled:
| situation | what to do |
|---|---|
| converged, stable | the adjoint (§2) is exact |
| converged, unstable (saddle or maximum of ) | do not backpropagate; the point is not in the relation. Restart inference, or anchor with |
| not converged, residual small | continue iterating from the current state (warm start), then backpropagate |
| not converged, budget exhausted | backpropagate through the unrolled last steps (exact for the truncated map), or use Jacobian-free backpropagation (replace by the identity, as in JFB) as a descent direction |
| no root at all (input outside the relation) | inference returns the nearest ridge point; the residual reports the distance |
The loss-side question (“what if inference increased the energy?”) does not arise for the solver here: Newton steps are only taken where , with backtracking on . Elsewhere the step is a descent step on when the field is a gradient. A learned non-conservative field has no energy to decrease, only a residual. That is why convergence is defined on and not on .
Observed in training (§7): the first prototype dropped about 60% of solves as non-converged, and its gradients blew the model up until those were excluded. With the stability guard and the Newton/descent switch, the package’s solver dropped none.
7. Learning a relation by backpropagating through inference
The experiment that closes the loop. Start from the circle, a ring of 64 mixture components whose means are . Pairs come from a different curve, . Train by Adam on , where is the inferred output, with gradients from the adjoint only and no denoising loss at all.
| before | after 150 epochs | |
|---|---|---|
| training loss (mean squared error) | ||
| test RMSE, warm start | 0.59 | 0.012 |
| test RMSE, cold start | 0.59 | 0.26 |
| dropped solves | — | 0 |
The cold-start number is one test point () landing on the upper arc of the old circle, which training never touched. Bilevel training reshapes the branch that inference visits and leaves the others. An implicit learner trained this way is a relation with several branches, and inference returns the branch of its starting basin. To fit a whole relation, either train generatively as well (the statistical-game semantics of The Implicit Diffusion Factor as a Statistical Game), or anchor the output () at an amortised guess.
8. Where AD is needed, and where it is not
| quantity | exact score (closed form) | learned network |
|---|---|---|
| forward pass | forward pass | |
| (inference Newton steps, adjoint solve) | closed form | input JVP/VJP, or finite differences for small |
| (parameter gradient) | closed form | reverse-mode AD: one VJP per node |
VariationalDiffusion has no AD dependency of its own: epsilon_jacobian has a
finite-difference fallback, and epsilon_vjp_params is an interface method that closed-form
predictors implement exactly. A Lux network gets both from the backend named in its predictor,
NoisePredictor(…; ad = AutoZygote()) (or Enzyme, ForwardDiff, Mooncake, Reactant), through a
package extension (backends). A trained 5k-parameter MLP on the circle reproduces the
finite-difference derivatives of this section to 8 digits (examples/circle_mlp.jl).
Compare RED-Diff as a Statistical Game §3, where the stop-gradient was “the difference
between a package with an AD dependency and one without”. Training through inference brings
back exactly one VJP per node, no more.
9. Backpropagating at higher noise levels than inference
An open idea, tried once (for ProxDM, below). Inference may stop at a lowest noise level : a sampler run
only from down to , or, in the implicit setting, field nodes only on
(field_nodes(…; levels)). The question is whether the backward pass should use higher
levels .
It does not have to. The adjoint with the forward’s own nodes is the exact gradient of the truncated problem, the one smoothed at level ; that is just §2 with the same model.
It may want to, for two reasons:
- Conditioning. Across the data manifold the Jacobian of the score grows like , so near small the adjoint solve is stiff, and the gradient is large and dominated by the network’s errors.
- Where the state lives. A state from a truncated forward pass sits about off the clean manifold; evaluating the network there at levels below is out of distribution. So the backward pass should never go lower than , and going higher is a regulariser.
The price is bias. With the forward’s Jacobian and the one at the higher levels, the surrogate adjoint is instead of : the gradient of a more smoothed problem than the one solved. It stays a descent direction while is close enough to the identity, the same kind of condition under which JFB (replacing by , §6) is a descent direction. Graduated smoothing is known to help non-convex optimisation (Gaussian homotopy, Mobahi & Fisher 2015), so a biased, smoother gradient might train better; that is the hypothesis.
Related work. Nothing found that does exactly this (forward at , adjoint at , and a study of ); every ingredient exists separately:
| idea | work | relation |
|---|---|---|
| inexact forward and inexact backward, with bounds | Pedregosa 2016; Grazzi et al. 2020 | hypergradient error ≲ forward error + backward error |
| approximate backward through an implicit layer | Fung et al. 2022 (JFB); Geng et al. 2021 (phantom gradients) | replaced by or a few unrolled steps: biased descent directions |
| backward at a noise level unrelated to the forward | Poole et al. 2023 (score distillation) | gradient at a random with the denoiser’s Jacobian dropped: the extreme case |
| backward through part of a sampler’s schedule | Clark et al. 2024 (DRaFT-K); Prabhudesai et al. 2023 (AlignProp) | the converse truncation: only the last, low-noise steps |
| optimising against a diffusion prior at a chosen or annealed | Graikos et al. 2022; Mardani et al. 2024 (RED-Diff) | the noise level as a smoothing knob for the gradient |
The experiment needs almost no new code. Build two ImplicitDiffusions on the same network,
one with levels on for the forward and one on for the backward. Solve
with the first and call implicit_pullback(m_backward, sol, …): it needs only sol.z and the
Jacobian, so it returns exactly the surrogate. Measure
- the cosine between surrogate and exact gradient as grows, and
- whether the training of §7 converges faster or more stably with .
Run for ProxDM (proxdm §8), where prox_pullback takes the noise level as its own
argument. On the §7 task with an exact prox: the surrogate’s cosine to the exact gradient is 0.98
at , 0.89 at and below 0.4 beyond. Over five
training sets, surrogates up to train about as well as the exact
gradient, none consistently better; at the bias stalls training in one run of
five. So no benefit, but also little cost over a wide range. The test had no conditioning
problem to fix (one free coordinate, an exact prox); the stiff case, a learned prox at small
with many free coordinates, is still untried, and so is the experiment for implicit_infer.
Related: Implicit Diffusion Learners, Inference Signatures, Deterministic Relaxation, The Implicit Diffusion Factor as a Statistical Game, DEQ as a Relation, Backpropagation by the Implicit Function Theorem, implicit, backends