Implicit inference and its backward pass for a diffusion model: a deterministic residual field, a solver that only reports roots it trusts, and an adjoint that turns one linear solve into gradients for parameters, inputs, anchors and precisions. The theory is in Implicit Diffusion Learners and Backpropagation through Implicit Inference.
Sources: code:
implicit.jl; the trust-region solver (§11): Nocedal & Wright, Numerical Optimization, 2nd ed., ch. 4 and 7; Steihaug 1983; the ridge residual (§5): Genovese et al. 2014, Ozertem & Erdogmus 2011 (the ridge as the set where the gradient projected on the Hessian’s normal eigenvectors vanishes; SCMS moves along the same projection of the mean-shift vector, here it is a Newton step)Bibliography: Nocedal & Wright 2006 · Steihaug 1983 · Genovese et al. 2014 · Ozertem & Erdogmus 2011 · Li et al. 2023
Theory (CT-ML wiki): Least Fixed Point · Statistical Game · Bayesian Lens · Reverse Derivative Category
1. What is in the file
| name | is |
|---|---|
FieldNodes, field_nodes, noisefree_nodes | the fixed that make the field deterministic |
ImplicitDiffusion | the relation: a NoisePredictor plus a node set plus λ; batched = true evaluates one noise level per network call |
prior_field, prior_jacobian, prior_vjp, prior_jvp | , the block of on chosen coordinates (coords), and |
implicit_residual | on the free coordinates |
implicit_infer → ImplicitSolution | a root, with residual, iters, converged, stable; linsolve = DenseSolve() or KrylovSolve() (krylov) |
implicit_pullback | the adjoint: (z₀ = …, ρ = …, ps = …) |
2. Design decisions
- Deterministic by construction. RED-Diff redraws every step; here the nodes are fixed, so inference is a function and the implicit function theorem applies.
field_nodesuses antithetic pairs so that the control variate cancels exactly. - Polarity as a precision vector, the same encoding as
precision_vector(f, polarity):Infis a hard clamp, imposed by projection, never as a penalty. - The solver trusts Newton only where it should. A Newton step is taken when the symmetric part of is positive definite, with backtracking on ; otherwise a fixed descent step along the field. Newton is attracted to every root, saddles and maxima included; restricting it to locally convex regions keeps it on stable roots.
- Two flags, not one.
converged(the residual is small) andstable() are separate, because a converged unstable root is on the zero set but not in the relation one wants. - The pullback refuses non-converged states. The IFT says nothing about a non-root, and silently returning a gradient there is what made the first prototype diverge.
- No AD dependency. The input Jacobian has a finite-difference fallback (
epsilon_jacobian); the parameter VJP is an interface method (epsilon_vjp_params) that closed-form predictors implement exactly and a Lux network gets from the AD backend in itsadfield (backends). A model with no parameters skips it.
3. Implementation difficulties
- The tolerance floor. The field of a narrow mixture at small has a floating-point noise floor around (it scales like ). An absolute tolerance of made a third of the solves “fail”; the tolerance is relative,
tol * (1 + ‖z‖). - The prototype’s solver. A version with a fixed descent step in non-convex regions and an energy line search dropped about 60% of solves during training, and its training stalled at RMSE 0.15. The package’s solver (Newton where convex, residual backtracking) dropped none and reached 0.012.
- Smoothing levels. With RED-Diff’s training range of noise levels, the deterministic field of a circle has its only root at the centre. The default
levelsstop at ; the trade-off is measured in Implicit Diffusion Learners §5. - Dense Jacobians. The block is formed densely by default, which is fine while few coordinates are free (§6) but not when a whole image is free.
linsolve = KrylovSolve()replaces it by GMRES on Jacobian-vector products in the Newton step and on vector-Jacobian products in the adjoint, and by a Lanczos test of stability (krylov).
4. How it is tested
test/runtests.jl, against the closed-form mixture of analytic and the Gaussian oracle:
- the field equals and its Jacobian is symmetric;
- both branches of the circle are found, and they merge near the branch point;
- the smoothing bias is monotone in the levels and matches ;
- noise-free roots are Tweedie fixed points;
- the Gaussian root and its derivative match the closed form;
- every adjoint cotangent matches finite differences of the full solve;
- training through inference reduces the loss by more than 50×;
- the Krylov solver, adjoint, Laplace marginals and mixtures agree with the dense ones, and
prior_jvpequals on every backend; - Zygote through
implicit_infer(itsrrule) equalsimplicit_pullbackand finite differences; - the free-coordinate block equals the slice of the full Jacobian and
prior_vjpequals , for the closed form, finite differences, Zygote, ForwardDiff, Enzyme, energy networks, products and (separate suite) Reactant; batched and unbatched nodes give the same field, Jacobian, answer and gradient; implicit_rootsfinds both branches of the circle, and the Laplace covariance of a Gaussian query equals the closed form .
5. All answers, and how sure (answers.jl)
Two functions build on implicit_infer without changing it.
implicit_roots solves from z₀ and from perturbed starts (normal on the free coordinates,
the clamped ones fixed), keeps converged and stable answers, and removes duplicates. For an
energy-parametrised predictor the answers are ordered by the query’s energy. On the circle,
” given ” returns both branches; near they merge into one. It finds
what its starts reach: no guarantee of completeness
(Open Problems in Implicit Diffusion Learning T1).
implicit_laplace inverts the symmetric part of , the residual’s Jacobian on the
free coordinates at the answer, which is the curvature of the query’s energy. The subtle part
is the units. With and ,
so the prior term is a sum of smoothed negative log-densities with weights
. They sum to one exactly when ,
density_lambda(schedule, nodes), and then the energy is a proper negative log-density in
the units of the clamp’s Gaussian likelihood, and the covariance is in data units. The
linear terms come from the noise draws and cancel for antithetic nodes. For queries with only
hard inputs and free outputs, λ scales the curvature and leaves the answers unchanged.
On the circle with this λ, the standard deviation of is 0.135 at (the smoothed ring’s width: data width 0.05 combined with the noise levels gives about 0.12), 0.17 at (the vertical line crosses the ring obliquely), and it grows towards the branch points (0.31, then 0.41) and off the circle (0.53). Calibration is to the density smoothed at the field’s noise levels, so it is wider than the data’s own spread, the same smoothing that biases the answers inwards (Implicit Diffusion Learners §5).
laplace_belief returns that covariance as a GaussianBelief (marginalised onto chosen
free coordinates), and implicit_mixture combines all answers into a MixtureBelief. Its
weights are Laplace estimates of each answer’s probability mass,
, with the query’s energy. That needs an
energy: energy networks have one, and so does the closed-form mixture, whose energy is the
same formula with in closed form (so ordering in implicit_roots uses it
too). With on a circle whose upper half carries three times the data, the upper
answer gets weight 0.78. The unweighted circle gives 0.54 rather than 0.5, because the field’s
random noise draws are not symmetric; the ratio of the two odds is 3.0. For plain ε-networks
the weights are equal.
coverage (coverage.jl) says whether an answer lies where the model has seen data: the
distance to the nearest data point, the denoiser’s reconstruction error and the energy, each as
a p-value against held-out points. A clamp off the relation still converges, and is flagged
there (coverage).
round_trip (F3) asks the same question of the model alone, in data units. It reverses the
query: the hard-clamped coordinates are freed, every other one is clamped at the answer, and
implicit_infer solves back from the answer. The result is , how far the
clamps move. An answer on the relation is already a root of the reversed query and stays; a
query off the relation ( on the circle) has its root on the smoothed density’s ridge,
and the reversed query walks back onto the relation. For a radially symmetric density the
modes of both slices lie on the same mode circle, so on the closed-form ring the distance of an
answer on the relation is zero up to the field’s discreteness, and from it is
with the smoothed ring’s mode radius (test “round trip: …”). A reversed solve that
does not converge to a stable root gives Inf.
It costs one more solve per answer and needs no data and no reference set, unlike coverage.
What it does not see: an answer on a wrong branch is on the relation and passes (T1), and with
a smoothing bias larger than the tolerance an answer on the relation fails. It also fails answers
where the relation runs parallel to the clamped coordinates (the top of the circle for ” given
”): the reversed query is ill-conditioned there and does not return to the clamp, so the check
asks for local invertibility in the reverse direction as well as membership. Its effect on the
calibration benchmark is in examples/calibration.md.
ridge_residual (F3) tests membership alone, from the field at the answer, with no further
solve. The relation is the ridge of the smoothed density (Genovese et al. 2014; Ozertem & Erdogmus
2011): across it the field pulls back, along it the field may still
drift where the density varies along the relation. With the symmetric part of the full Jacobian
(clamped coordinates included), its eigenvectors of the largest
eigenvalues ( the codimension, by default the number of free coordinates) and the
field, the normal Newton step
is the first-order displacement to the ridge, and its distance in data units. The
tangential part of is dropped, and that is the difference from the full score’s norm. For a
Gaussian the field is affine with a symmetric Jacobian that shares the covariance’s eigenvectors,
so the step is exact: the distance to the major axis through the root (codimension 1), or to the
root itself (codimension 2), to ; the weight λ scales and alike and cancels (test
“ridge residual: …”). On the closed-form ring the answers for and are within
of the ridge with a radial normal, the projection from is away to
, and at , the top, the round trip moves the clamp by about 0.2 where the ridge
residual passes. When a normal curvature is not positive the field does not pull back across, and
the distance is Inf (the ring’s centre). The result also carries the eigengap between normal and
tangential curvatures: where it is small (a crossing of two branches, an isotropic blob) the normal
directions, and so the distance, are not well determined.
What it does not do: the Jacobian is formed in full, one derivative pass per coordinate and level,
so there is no matrix-free version for a state as large as MNIST’s (query_report skips it with
KrylovSolve); the codimension must be known or assumed; a non-conservative field (a trained
ε-network) is judged by its Jacobian’s symmetric part; and, like the round trip, an answer on a
wrong branch passes. Its effect on the calibration benchmark is in examples/calibration.md.
What is still missing: λ chosen for calibration is not always the λ one wants for balancing soft evidence.
6. Cost: free-coordinate blocks and batched nodes
Every consumer of the Jacobian needs only a block of it: the solver and its stability check need
, the Laplace covariance needs , and the adjoint needs and
. So prior_jacobian(m, z, ps, st; coords = free) computes only , and
prior_vjp computes (hence ) without forming . The cost
then grows with the number of free coordinates, not with the dimension:
| backend | the block , per noise level | |
|---|---|---|
| none (finite differences) | forward passes along the free coordinates | from the per-node Jacobian (slow; small only) |
GaussianMixtureEps | closed form, restricted | closed form |
| DifferentiationInterface | one Jacobian of , a map : JVPs or VJPs | one input VJP |
| energy networks | the Hessian of : Hessian-vector products | one HVP |
| Reactant | compiled input VJPs | one compiled VJP |
All columns move together with (), so one derivative of the weighted sum over a noise level’s nodes covers all of them.
Batching. With batched = true the nodes of one noise level go through the network as the
columns of one matrix, for the field, the block, the VJPs and the parameter cotangent: 8 calls
instead of 64 with the default nodes. It is opt-in because the predictor must accept a matrix at
a scalar time; the closed-form mixture does not.
Measured (an MLP , 64 nodes, Zygote, CPU), a query with 10 of 794 coordinates free, as in the MNIST classification of MNIST in Both Directions:
| time | |
|---|---|
| full Jacobian, as every solver iteration used to compute it | ≈ 26 s (batched; unbatched is slower) |
| the block, unbatched | 0.74 s |
| the block, batched | 0.04 s |
| the field, unbatched / batched | 3.1 ms / 1.0 ms |
With ForwardDiff the block costs 0.04 s either way: forward mode already pays per free coordinate.
Many free coordinates (the image generated, the label clamped: 784 of 794 free) make the block
itself the cost: one dense block takes 33 s. With KrylovSolve a whole
implicit_infer takes 1.1 s (two Newton iterations), the adjoint 0.24 s and a Laplace marginal on
ten coordinates 0.8 s; one Jacobian-vector product costs 3.2 ms and one vector-Jacobian product
5.7 ms (krylov §4).
7. Many queries at once (many.jl)
implicit_infer_many(m, Z₀, ρ, ps, st) solves the columns of Z₀ as separate queries with
one polarity. Each query keeps its own Newton or descent step, backtracking, stability test and
stopping rule, and leaves the batch when it converges. What is shared is the network: per
iteration, one call per noise level evaluates the field of all unfinished queries. The
free-coordinate blocks of all of them come from one more call per level, which stacks the
directions as copies of the inputs side by side: JVPs in general, VJPs (rows)
when the predictor has an AD backend. The blocks are dense, so this suits few free coordinates,
which is classification’s shape.
The answers are those of implicit_infer, query by query: equal to on the circle,
with the same iteration counts, and to for a Lux network with every backend, batched or
not (tests). On the CPU the gain is small: 1.9 s against 2.6 s one by one, for 100 queries with finite
differences, where the network’s cost dominates. On a GPU it is the difference between 24 ms and
1.8 ms per query and iteration (energy_network §8).
8. Classification: categorical answers
implicit_classify(m, z₀, ρ, ps, st; label, classes) makes the label coordinates free, starts
the solver from every class code (the columns of classes, e.g. one-hot vectors), and reads the
implicit_mixture on the label coordinates as a class with categorical_answer. Each root goes
to its nearest code, and the Laplace masses of the roots of a class are summed. Roots at least
half the smallest distance between two codes away from every code are reported as unassigned
mass, not dropped and not renormalised away. The answer keeps the mixture it was read from. With
a matrix of queries, implicit_classify solves all queries from all codes in one
implicit_infer_many, and batches the Jacobian blocks and energies of the roots as well.
Measured on three classes in the plane with one-hot labels (closed-form mixtures, width
0.15, density_lambda), against the exact posterior of the data:
| answer | exact | |
|---|---|---|
| a class centre | 1.000 | 1.000 |
| on the boundary between two classes | 0.500 / 0.500 | 0.500 / 0.500 |
| 0.02 off the boundary | 0.713 / 0.287 | 0.744 / 0.256 |
| 0.04 off | 0.860 / 0.140 | 0.895 / 0.105 |
| 0.06 off | 0.939 / 0.061 | 0.961 / 0.039 |
Within 0.04 everywhere, and slightly less confident than the data. This is the smoothing: the
answer is the posterior of the density smoothed at the field’s noise levels, which is wider
than the data’s. A query far from every class () is still answered with certainty
(class 3, the nearest). Classification inherits the extrapolation problem, and coverage is the
flag for it (coverage).
9. Conditional fields: block levels
With a predictor with noise blocks (predictor §6), ImplicitDiffusion(pred, nodes; levels)
says, per block, whether it is annealed with the nodes (nothing) or held at a fixed level.
conditional_levels(pred, ρ) holds every fully clamped block at tmin: the network sees the
clamp clean at every node, and the field on the free coordinates is the smoothed conditional
score instead of the joint score at a common level. The node input is
with , per coordinate. The weight of coordinate at a
node of level is : RED-Diff’s on the annealed
blocks, and chosen so that the field of an energy network stays the gradient of
which implicit_energy returns. The Jacobian block on the free coordinates gains the factor
; the VJP scales the cotangent by before the network and by after.
Without levels every formula reduces to the scalar one, and the tests check that a blocked
predictor with every block annealed gives the unblocked relation.
What it changes: the bias against conditioning (T2). On the unit circle (48 narrow components), query given , 19 values of in , the radius of the stable answers (exact: 1):
| field levels | joint field (x smoothed too) | x held clean |
|---|---|---|
| 0.002–0.05 | 0.978 ± 0.008 (18/19 stable) | 0.993 ± 0.011 (19/19) |
| 0.05–0.2 | 0.637 ± 0.135 (18/19) | 0.775 ± 0.048 (19/19) |
Smoothing as well pulls the joint answer inwards; holding the clamp clean leaves only the smoothing in . It does not remove the bias, which now comes from the free coordinates alone. Tested: field = , Jacobian against finite differences, adjoint against finite differences in the clamp and in a mixture mean, batched against unbatched, many queries against single ones, a Lux energy network on JLArrays.
10. Single roots, tolerances, continuation
Found on MNIST (MNIST in Both Directions §6; Open Problems I13):
- Probabilities from a single root. When a classification query has one stable root, the
Laplace mixture gives the class probabilities 0 and 1.
categorical_answer(mix, classes; temperature = τ, cov_scale = κ)instead reads each root’s Gaussian as evidence about the code: . A root between two codes splits its mass between them.calibrate_temperature(answers, classes, truth)fits τ and κ by maximum likelihood on held-out answers (alternating golden sections over and ). Tests: for answers drawn as it recovers , and for a covariance twice too wide in standard deviation . On MNIST the Laplace covariance is too wide (κ = 0.45), and with both parameters the probabilities have a calibration error of 0.009 (MNIST in Both Directions §7).implicit_classifypasses both on. - Unassigned mass as a coverage flag.
coverage(ref, z; unassigned)returns the mass and setsflaggedwhen or at least half the answer lies between the codes. - Tolerances that follow the arithmetic.
toldefaults todefault_tolerance(m, z, ps), for the type the field is computed in: the parameters’ type on a device (Float32 on a GPU gives about ), and on the CPU the promotion of parameters and state (Float32 weights against a Float64 state compute in Float64, so ). A Float32 field’s rounding floor is 10 to 15 relative, with 10 free coordinates and with 784 (§11); the first version, , sat on it, and solves stalled just above. - Continuation in the noise level.
implicit_infer(…; continuation = (8, 3))solves first with every node’s level multiplied by 8 (capped at 1), then by 3, each from the previous answer, then the actual relation. A smoother field has fewer, wider basins. With Newton-or-descent it is not a general cure: on the circle a high level smooths both branches into one blob centred on the origin, and the continuation stops at the symmetric point between them (from , the plain solve finds the branch in 5 iterations; the continuation ends at , unstable). With the trust region (§11) it walks off that point onto a branch. It is for starts far from every basin with a single plausible answer, such as a clamped label and a blank image: on MNIST, 3–4 of 10 modes with 784 free pixels converge with it and none without, at five to nine times the cost (MNIST in Both Directions §7).
11. A trust-region solver
Newton-or-descent takes a Newton step where is positive definite and a fixed
descent step along the residual elsewhere. Far from a basin, and in non-convex regions, the
descent steps are too short and the Newton steps are not taken, so the iteration stalls.
implicit_infer(…; method = :trust_region) is a trust-region method on a merit function
(Nocedal & Wright 2006, ch. 4):
- with an energy, the query energy , whose gradient is the residual and whose Hessian is . Every accepted step lowers , so the iteration cannot settle on a saddle or a maximum; negative curvature is a direction to move in, not a reason to fall back;
- without one, with the Gauss–Newton model (Levenberg–Marquardt), which
can stop at a minimum of the residual that is not a root;
convergedthen says so.
Each step minimises the quadratic model in a ball of radius : exactly with DenseSolve
(the eigendecomposition and a one-dimensional search for the shift, with the hard case), by
Steihaug’s truncated conjugate gradients (Steihaug 1983) with KrylovSolve (Hessian-vector products only).
starts at and is halved or doubled by the ratio of actual to predicted decrease.
Near the answer that ratio drowns in the rounding of , computed in the network’s precision;
there a smaller residual decides instead.
method = :auto, the default, uses it for relations with an energy (energy networks, mixtures,
products of them) and Newton-or-descent otherwise. implicit_infer_many has the same method for
many queries: each query keeps its own radius and acceptance test, and the residuals, Jacobian
blocks and energies of all unfinished queries, and of their trial points, are evaluated together.
Each query’s answer and iteration count are those of implicit_infer (a test, on the circle from
28 starts including far ones).
Measured. On the circle (y given x, 20 values of and 8 starts each, 50 iterations at most):
| converged and stable | mean iterations | |
|---|---|---|
| Newton-or-descent | 122/160 | 18.3 |
| trust region (dense or Krylov, identical) | 160/160 | 6.6 |
On MNIST, modes with the label held and 784 free pixels (MNIST in Both Directions §7, at most 50 iterations):
| start | method | converged and stable | median residual | iterations | time per mode |
|---|---|---|---|---|---|
| blank image | Newton-or-descent | 0/10 | 2.9 | 50 | 22 s |
| blank image | trust region | 10/10 | 16 | 6 s | |
| a sample | Newton-or-descent | 0/10 | 16 | 50 | 19 s |
| a sample | trust region | 9/10 | 37 | 6 s |
Continuation in the noise level with Newton-or-descent reached 3–4 of 10 at 120–220 s per mode; the trust region needs neither it nor the time.
On easy queries it costs a little more. MNIST classification (one start per query, batches of
1000, 1000 tuning images) converges for every query with either method, with the same
accuracy; the trust region takes 15.7–67.7 ms per query against 13.1–66.4 ms for Newton-or-descent
(2–20% more, for the energy it evaluates at each trial point). For large batches of easy
queries method = :newton is the faster choice. Getting there needed the tolerance fix of §10:
the solves first stalled at residuals of , just above a tolerance of .
12. What a query costs (cost.jl)
Root finding and the diffusion classifier (Li et al. 2023) read the same network. Counting the network passes each makes per query, and weighting them by their measured cost per column, predicts which is cheaper:
with starts, iterations, field nodes, free coordinates (one input derivative
each per node, for the Jacobian block), classes and noise draws; the trust region adds
the energy at each trial point, Newton-or-descent a trial residual instead.
pass_costs(pred, X, t, ps, st; directions) measures , and at the batch width
the solvers use; root_cost, classifier_cost and break_even_classes evaluate the formulas.
Measured on MNIST (block model, image held clean, levels 0.05–0.2, 16 nodes, 1000 tuning images, RTX 3080): per column = 2.9 µs, = 1.1 µs, = 15.1 µs (a Hessian-vector product, forward over reverse, about five ε evaluations).
| measured | predicted | accuracy | |
|---|---|---|---|
| root finding, Newton-or-descent (3.0 iterations) | 15.1 ms | 10.0 ms | 95.0% |
| root finding, trust region (3.8 iterations) | 18.5 ms | 11.9 ms | 95.0% |
| diffusion classifier, 10 × 32 draws | 3.1 ms | 2.0 ms | 93.9% |
| diffusion classifier, 10 × 128 | 10.7 ms | 8.1 ms | 94.2% |
| diffusion classifier, 10 × 256 | 21.5 ms | 16.2 ms | 93.7% |
The predictions are 1.3 to 1.6 times too low: host-side work (assembling node inputs, transfers, narrower calls as queries leave the batch) is not counted. It is similar for both methods, so the ratio is right within 10% (5.0 predicted against 4.9 measured for Newton, 6.0 against 6.0 for the trust region, at 32 draws), and the ratio decides which to use.
What it says.
- With one-hot labels, : root finding’s cost grows with the classes as fast as the classifier’s, with a larger constant ( against per class). It is never cheaper; here 5–6 times dearer at 32 draws.
- With a label code of fixed dimension (an embedding instead of one-hot), root finding’s cost is fixed and the classifier’s grows with . At these units (the classifier called in wide batches, 2.9 µs per ε; in chunks of 100 as measured above it costs twice that) it is cheaper above classes (Newton-or-descent; with the trust region): about 45 classes for a 4-dimensional code, 110 for a 10-dimensional one.
- The classifier’s accuracy did not grow with more draws (93.7–94.2% from 32 to 256), one point below root finding’s. Where accuracy matters, the comparison is not at equal accuracy at all.
- The derivative passes dominate: is about 50 times here. Halving the nodes halves the cost; Newton-or-descent is cheaper than the trust region on easy queries; a cheaper Jacobian block (fewer directions by Krylov, or a reverse-mode Hessian) is where root finding gains most.
Related: analytic, reddiff, factor, energy_network, krylov, sampler, Deterministic Relaxation, Inference Signatures, The Implicit Diffusion Factor as a Statistical Game
Singular Jacobians in the pullback; non-conservative Laplace (2026-11-04, LP-023)
- The dense
implicit_pullbacksolve refuses a free-block Jacobian that is singular to working precision (, on a host copy; skipped for AD number types). At a fold or branch point the implicit function theorem does not apply, and the solve returned Inf/NaN. Tikhonov damping, which one review proposed, would return a gradient of a different problem silently. The Krylov path warns when GMRES stops short, as before. implicit_laplaceof a field that is not a gradient (a noise predictor that is not an energy network) returns the inverse of , which is then a local-curvature measure, not a posterior covariance; the docstring says so and a warning fires when .
Tests: test/review_singular.jl (a stub relation supplies the Jacobian).