implementation

Implicit inference and its backward pass for a diffusion model: a deterministic residual field, a solver that only reports roots it trusts, and an adjoint that turns one linear solve into gradients for parameters, inputs, anchors and precisions. The theory is in Implicit Diffusion Learners and Backpropagation through Implicit Inference.

Sources: code: implicit.jl; the trust-region solver (§11): Nocedal & Wright, Numerical Optimization, 2nd ed., ch. 4 and 7; Steihaug 1983; the ridge residual (§5): Genovese et al. 2014, Ozertem & Erdogmus 2011 (the ridge as the set where the gradient projected on the Hessian’s normal eigenvectors vanishes; SCMS moves along the same projection of the mean-shift vector, here it is a Newton step)

Bibliography: Nocedal & Wright 2006 · Steihaug 1983 · Genovese et al. 2014 · Ozertem & Erdogmus 2011 · Li et al. 2023

Theory (CT-ML wiki): Least Fixed Point · Statistical Game · Bayesian Lens · Reverse Derivative Category

1. What is in the file

nameis
FieldNodes, field_nodes, noisefree_nodesthe fixed that make the field deterministic
ImplicitDiffusionthe relation: a NoisePredictor plus a node set plus λ; batched = true evaluates one noise level per network call
prior_field, prior_jacobian, prior_vjp, prior_jvp, the block of on chosen coordinates (coords), and
implicit_residual on the free coordinates
implicit_infer → ImplicitSolutiona root, with residual, iters, converged, stable; linsolve = DenseSolve() or KrylovSolve() (krylov)
implicit_pullbackthe adjoint: (z₀ = …, ρ = …, ps = …)

2. Design decisions

  • Deterministic by construction. RED-Diff redraws every step; here the nodes are fixed, so inference is a function and the implicit function theorem applies. field_nodes uses antithetic pairs so that the control variate cancels exactly.
  • Polarity as a precision vector, the same encoding as precision_vector(f, polarity): Inf is a hard clamp, imposed by projection, never as a penalty.
  • The solver trusts Newton only where it should. A Newton step is taken when the symmetric part of is positive definite, with backtracking on ; otherwise a fixed descent step along the field. Newton is attracted to every root, saddles and maxima included; restricting it to locally convex regions keeps it on stable roots.
  • Two flags, not one. converged (the residual is small) and stable () are separate, because a converged unstable root is on the zero set but not in the relation one wants.
  • The pullback refuses non-converged states. The IFT says nothing about a non-root, and silently returning a gradient there is what made the first prototype diverge.
  • No AD dependency. The input Jacobian has a finite-difference fallback (epsilon_jacobian); the parameter VJP is an interface method (epsilon_vjp_params) that closed-form predictors implement exactly and a Lux network gets from the AD backend in its ad field (backends). A model with no parameters skips it.

3. Implementation difficulties

  1. The tolerance floor. The field of a narrow mixture at small has a floating-point noise floor around (it scales like ). An absolute tolerance of made a third of the solves “fail”; the tolerance is relative, tol * (1 + ‖z‖).
  2. The prototype’s solver. A version with a fixed descent step in non-convex regions and an energy line search dropped about 60% of solves during training, and its training stalled at RMSE 0.15. The package’s solver (Newton where convex, residual backtracking) dropped none and reached 0.012.
  3. Smoothing levels. With RED-Diff’s training range of noise levels, the deterministic field of a circle has its only root at the centre. The default levels stop at ; the trade-off is measured in Implicit Diffusion Learners §5.
  4. Dense Jacobians. The block is formed densely by default, which is fine while few coordinates are free (§6) but not when a whole image is free. linsolve = KrylovSolve() replaces it by GMRES on Jacobian-vector products in the Newton step and on vector-Jacobian products in the adjoint, and by a Lanczos test of stability (krylov).

4. How it is tested

test/runtests.jl, against the closed-form mixture of analytic and the Gaussian oracle:

  • the field equals and its Jacobian is symmetric;
  • both branches of the circle are found, and they merge near the branch point;
  • the smoothing bias is monotone in the levels and matches ;
  • noise-free roots are Tweedie fixed points;
  • the Gaussian root and its derivative match the closed form;
  • every adjoint cotangent matches finite differences of the full solve;
  • training through inference reduces the loss by more than 50×;
  • the Krylov solver, adjoint, Laplace marginals and mixtures agree with the dense ones, and prior_jvp equals on every backend;
  • Zygote through implicit_infer (its rrule) equals implicit_pullback and finite differences;
  • the free-coordinate block equals the slice of the full Jacobian and prior_vjp equals , for the closed form, finite differences, Zygote, ForwardDiff, Enzyme, energy networks, products and (separate suite) Reactant; batched and unbatched nodes give the same field, Jacobian, answer and gradient;
  • implicit_roots finds both branches of the circle, and the Laplace covariance of a Gaussian query equals the closed form .

5. All answers, and how sure (answers.jl)

Two functions build on implicit_infer without changing it.

implicit_roots solves from z₀ and from perturbed starts (normal on the free coordinates, the clamped ones fixed), keeps converged and stable answers, and removes duplicates. For an energy-parametrised predictor the answers are ordered by the query’s energy. On the circle, ” given ” returns both branches; near they merge into one. It finds what its starts reach: no guarantee of completeness (Open Problems in Implicit Diffusion Learning T1).

implicit_laplace inverts the symmetric part of , the residual’s Jacobian on the free coordinates at the answer, which is the curvature of the query’s energy. The subtle part is the units. With and ,

so the prior term is a sum of smoothed negative log-densities with weights . They sum to one exactly when , density_lambda(schedule, nodes), and then the energy is a proper negative log-density in the units of the clamp’s Gaussian likelihood, and the covariance is in data units. The linear terms come from the noise draws and cancel for antithetic nodes. For queries with only hard inputs and free outputs, λ scales the curvature and leaves the answers unchanged.

On the circle with this λ, the standard deviation of is 0.135 at (the smoothed ring’s width: data width 0.05 combined with the noise levels gives about 0.12), 0.17 at (the vertical line crosses the ring obliquely), and it grows towards the branch points (0.31, then 0.41) and off the circle (0.53). Calibration is to the density smoothed at the field’s noise levels, so it is wider than the data’s own spread, the same smoothing that biases the answers inwards (Implicit Diffusion Learners §5).

laplace_belief returns that covariance as a GaussianBelief (marginalised onto chosen free coordinates), and implicit_mixture combines all answers into a MixtureBelief. Its weights are Laplace estimates of each answer’s probability mass, , with the query’s energy. That needs an energy: energy networks have one, and so does the closed-form mixture, whose energy is the same formula with in closed form (so ordering in implicit_roots uses it too). With on a circle whose upper half carries three times the data, the upper answer gets weight 0.78. The unweighted circle gives 0.54 rather than 0.5, because the field’s random noise draws are not symmetric; the ratio of the two odds is 3.0. For plain ε-networks the weights are equal.

coverage (coverage.jl) says whether an answer lies where the model has seen data: the distance to the nearest data point, the denoiser’s reconstruction error and the energy, each as a p-value against held-out points. A clamp off the relation still converges, and is flagged there (coverage).

round_trip (F3) asks the same question of the model alone, in data units. It reverses the query: the hard-clamped coordinates are freed, every other one is clamped at the answer, and implicit_infer solves back from the answer. The result is , how far the clamps move. An answer on the relation is already a root of the reversed query and stays; a query off the relation ( on the circle) has its root on the smoothed density’s ridge, and the reversed query walks back onto the relation. For a radially symmetric density the modes of both slices lie on the same mode circle, so on the closed-form ring the distance of an answer on the relation is zero up to the field’s discreteness, and from it is with the smoothed ring’s mode radius (test “round trip: …”). A reversed solve that does not converge to a stable root gives Inf.

It costs one more solve per answer and needs no data and no reference set, unlike coverage. What it does not see: an answer on a wrong branch is on the relation and passes (T1), and with a smoothing bias larger than the tolerance an answer on the relation fails. It also fails answers where the relation runs parallel to the clamped coordinates (the top of the circle for ” given ”): the reversed query is ill-conditioned there and does not return to the clamp, so the check asks for local invertibility in the reverse direction as well as membership. Its effect on the calibration benchmark is in examples/calibration.md.

ridge_residual (F3) tests membership alone, from the field at the answer, with no further solve. The relation is the ridge of the smoothed density (Genovese et al. 2014; Ozertem & Erdogmus 2011): across it the field pulls back, along it the field may still drift where the density varies along the relation. With the symmetric part of the full Jacobian (clamped coordinates included), its eigenvectors of the largest eigenvalues ( the codimension, by default the number of free coordinates) and the field, the normal Newton step

is the first-order displacement to the ridge, and its distance in data units. The tangential part of is dropped, and that is the difference from the full score’s norm. For a Gaussian the field is affine with a symmetric Jacobian that shares the covariance’s eigenvectors, so the step is exact: the distance to the major axis through the root (codimension 1), or to the root itself (codimension 2), to ; the weight λ scales and alike and cancels (test “ridge residual: …”). On the closed-form ring the answers for and are within of the ridge with a radial normal, the projection from is away to , and at , the top, the round trip moves the clamp by about 0.2 where the ridge residual passes. When a normal curvature is not positive the field does not pull back across, and the distance is Inf (the ring’s centre). The result also carries the eigengap between normal and tangential curvatures: where it is small (a crossing of two branches, an isotropic blob) the normal directions, and so the distance, are not well determined.

What it does not do: the Jacobian is formed in full, one derivative pass per coordinate and level, so there is no matrix-free version for a state as large as MNIST’s (query_report skips it with KrylovSolve); the codimension must be known or assumed; a non-conservative field (a trained ε-network) is judged by its Jacobian’s symmetric part; and, like the round trip, an answer on a wrong branch passes. Its effect on the calibration benchmark is in examples/calibration.md.

What is still missing: λ chosen for calibration is not always the λ one wants for balancing soft evidence.

6. Cost: free-coordinate blocks and batched nodes

Every consumer of the Jacobian needs only a block of it: the solver and its stability check need , the Laplace covariance needs , and the adjoint needs and . So prior_jacobian(m, z, ps, st; coords = free) computes only , and prior_vjp computes (hence ) without forming . The cost then grows with the number of free coordinates, not with the dimension:

backendthe block , per noise level
none (finite differences) forward passes along the free coordinatesfrom the per-node Jacobian (slow; small only)
GaussianMixtureEpsclosed form, restrictedclosed form
DifferentiationInterfaceone Jacobian of , a map : JVPs or VJPsone input VJP
energy networksthe Hessian of : Hessian-vector productsone HVP
Reactant compiled input VJPsone compiled VJP

All columns move together with (), so one derivative of the weighted sum over a noise level’s nodes covers all of them.

Batching. With batched = true the nodes of one noise level go through the network as the columns of one matrix, for the field, the block, the VJPs and the parameter cotangent: 8 calls instead of 64 with the default nodes. It is opt-in because the predictor must accept a matrix at a scalar time; the closed-form mixture does not.

Measured (an MLP , 64 nodes, Zygote, CPU), a query with 10 of 794 coordinates free, as in the MNIST classification of MNIST in Both Directions:

time
full Jacobian, as every solver iteration used to compute it≈ 26 s (batched; unbatched is slower)
the block, unbatched0.74 s
the block, batched0.04 s
the field, unbatched / batched3.1 ms / 1.0 ms

With ForwardDiff the block costs 0.04 s either way: forward mode already pays per free coordinate.

Many free coordinates (the image generated, the label clamped: 784 of 794 free) make the block itself the cost: one dense block takes 33 s. With KrylovSolve a whole implicit_infer takes 1.1 s (two Newton iterations), the adjoint 0.24 s and a Laplace marginal on ten coordinates 0.8 s; one Jacobian-vector product costs 3.2 ms and one vector-Jacobian product 5.7 ms (krylov §4).

7. Many queries at once (many.jl)

implicit_infer_many(m, Z₀, ρ, ps, st) solves the columns of Z₀ as separate queries with one polarity. Each query keeps its own Newton or descent step, backtracking, stability test and stopping rule, and leaves the batch when it converges. What is shared is the network: per iteration, one call per noise level evaluates the field of all unfinished queries. The free-coordinate blocks of all of them come from one more call per level, which stacks the directions as copies of the inputs side by side: JVPs in general, VJPs (rows) when the predictor has an AD backend. The blocks are dense, so this suits few free coordinates, which is classification’s shape.

The answers are those of implicit_infer, query by query: equal to on the circle, with the same iteration counts, and to for a Lux network with every backend, batched or not (tests). On the CPU the gain is small: 1.9 s against 2.6 s one by one, for 100 queries with finite differences, where the network’s cost dominates. On a GPU it is the difference between 24 ms and 1.8 ms per query and iteration (energy_network §8).

8. Classification: categorical answers

implicit_classify(m, z₀, ρ, ps, st; label, classes) makes the label coordinates free, starts the solver from every class code (the columns of classes, e.g. one-hot vectors), and reads the implicit_mixture on the label coordinates as a class with categorical_answer. Each root goes to its nearest code, and the Laplace masses of the roots of a class are summed. Roots at least half the smallest distance between two codes away from every code are reported as unassigned mass, not dropped and not renormalised away. The answer keeps the mixture it was read from. With a matrix of queries, implicit_classify solves all queries from all codes in one implicit_infer_many, and batches the Jacobian blocks and energies of the roots as well.

Measured on three classes in the plane with one-hot labels (closed-form mixtures, width 0.15, density_lambda), against the exact posterior of the data:

answerexact
a class centre1.0001.000
on the boundary between two classes0.500 / 0.5000.500 / 0.500
0.02 off the boundary0.713 / 0.2870.744 / 0.256
0.04 off0.860 / 0.1400.895 / 0.105
0.06 off0.939 / 0.0610.961 / 0.039

Within 0.04 everywhere, and slightly less confident than the data. This is the smoothing: the answer is the posterior of the density smoothed at the field’s noise levels, which is wider than the data’s. A query far from every class () is still answered with certainty (class 3, the nearest). Classification inherits the extrapolation problem, and coverage is the flag for it (coverage).

9. Conditional fields: block levels

With a predictor with noise blocks (predictor §6), ImplicitDiffusion(pred, nodes; levels) says, per block, whether it is annealed with the nodes (nothing) or held at a fixed level. conditional_levels(pred, ρ) holds every fully clamped block at tmin: the network sees the clamp clean at every node, and the field on the free coordinates is the smoothed conditional score instead of the joint score at a common level. The node input is with , per coordinate. The weight of coordinate at a node of level is : RED-Diff’s on the annealed blocks, and chosen so that the field of an energy network stays the gradient of

which implicit_energy returns. The Jacobian block on the free coordinates gains the factor ; the VJP scales the cotangent by before the network and by after. Without levels every formula reduces to the scalar one, and the tests check that a blocked predictor with every block annealed gives the unblocked relation.

What it changes: the bias against conditioning (T2). On the unit circle (48 narrow components), query given , 19 values of in , the radius of the stable answers (exact: 1):

field levelsjoint field (x smoothed too)x held clean
0.002–0.050.978 ± 0.008 (18/19 stable)0.993 ± 0.011 (19/19)
0.05–0.20.637 ± 0.135 (18/19)0.775 ± 0.048 (19/19)

Smoothing as well pulls the joint answer inwards; holding the clamp clean leaves only the smoothing in . It does not remove the bias, which now comes from the free coordinates alone. Tested: field = , Jacobian against finite differences, adjoint against finite differences in the clamp and in a mixture mean, batched against unbatched, many queries against single ones, a Lux energy network on JLArrays.

10. Single roots, tolerances, continuation

Found on MNIST (MNIST in Both Directions §6; Open Problems I13):

  • Probabilities from a single root. When a classification query has one stable root, the Laplace mixture gives the class probabilities 0 and 1. categorical_answer(mix, classes; temperature = τ, cov_scale = κ) instead reads each root’s Gaussian as evidence about the code: . A root between two codes splits its mass between them. calibrate_temperature(answers, classes, truth) fits τ and κ by maximum likelihood on held-out answers (alternating golden sections over and ). Tests: for answers drawn as it recovers , and for a covariance twice too wide in standard deviation . On MNIST the Laplace covariance is too wide (κ = 0.45), and with both parameters the probabilities have a calibration error of 0.009 (MNIST in Both Directions §7). implicit_classify passes both on.
  • Unassigned mass as a coverage flag. coverage(ref, z; unassigned) returns the mass and sets flagged when or at least half the answer lies between the codes.
  • Tolerances that follow the arithmetic. tol defaults to default_tolerance(m, z, ps), for the type the field is computed in: the parameters’ type on a device (Float32 on a GPU gives about ), and on the CPU the promotion of parameters and state (Float32 weights against a Float64 state compute in Float64, so ). A Float32 field’s rounding floor is 10 to 15 relative, with 10 free coordinates and with 784 (§11); the first version, , sat on it, and solves stalled just above.
  • Continuation in the noise level. implicit_infer(…; continuation = (8, 3)) solves first with every node’s level multiplied by 8 (capped at 1), then by 3, each from the previous answer, then the actual relation. A smoother field has fewer, wider basins. With Newton-or-descent it is not a general cure: on the circle a high level smooths both branches into one blob centred on the origin, and the continuation stops at the symmetric point between them (from , the plain solve finds the branch in 5 iterations; the continuation ends at , unstable). With the trust region (§11) it walks off that point onto a branch. It is for starts far from every basin with a single plausible answer, such as a clamped label and a blank image: on MNIST, 3–4 of 10 modes with 784 free pixels converge with it and none without, at five to nine times the cost (MNIST in Both Directions §7).

11. A trust-region solver

Newton-or-descent takes a Newton step where is positive definite and a fixed descent step along the residual elsewhere. Far from a basin, and in non-convex regions, the descent steps are too short and the Newton steps are not taken, so the iteration stalls. implicit_infer(…; method = :trust_region) is a trust-region method on a merit function (Nocedal & Wright 2006, ch. 4):

  • with an energy, the query energy , whose gradient is the residual and whose Hessian is . Every accepted step lowers , so the iteration cannot settle on a saddle or a maximum; negative curvature is a direction to move in, not a reason to fall back;
  • without one, with the Gauss–Newton model (Levenberg–Marquardt), which can stop at a minimum of the residual that is not a root; converged then says so.

Each step minimises the quadratic model in a ball of radius : exactly with DenseSolve (the eigendecomposition and a one-dimensional search for the shift, with the hard case), by Steihaug’s truncated conjugate gradients (Steihaug 1983) with KrylovSolve (Hessian-vector products only). starts at and is halved or doubled by the ratio of actual to predicted decrease. Near the answer that ratio drowns in the rounding of , computed in the network’s precision; there a smaller residual decides instead.

method = :auto, the default, uses it for relations with an energy (energy networks, mixtures, products of them) and Newton-or-descent otherwise. implicit_infer_many has the same method for many queries: each query keeps its own radius and acceptance test, and the residuals, Jacobian blocks and energies of all unfinished queries, and of their trial points, are evaluated together. Each query’s answer and iteration count are those of implicit_infer (a test, on the circle from 28 starts including far ones).

Measured. On the circle (y given x, 20 values of and 8 starts each, 50 iterations at most):

converged and stablemean iterations
Newton-or-descent122/16018.3
trust region (dense or Krylov, identical)160/1606.6

On MNIST, modes with the label held and 784 free pixels (MNIST in Both Directions §7, at most 50 iterations):

startmethodconverged and stablemedian residualiterationstime per mode
blank imageNewton-or-descent0/102.95022 s
blank imagetrust region10/10166 s
a sampleNewton-or-descent0/10165019 s
a sampletrust region9/10376 s

Continuation in the noise level with Newton-or-descent reached 3–4 of 10 at 120–220 s per mode; the trust region needs neither it nor the time.

On easy queries it costs a little more. MNIST classification (one start per query, batches of 1000, 1000 tuning images) converges for every query with either method, with the same accuracy; the trust region takes 15.7–67.7 ms per query against 13.1–66.4 ms for Newton-or-descent (2–20% more, for the energy it evaluates at each trial point). For large batches of easy queries method = :newton is the faster choice. Getting there needed the tolerance fix of §10: the solves first stalled at residuals of , just above a tolerance of .

12. What a query costs (cost.jl)

Root finding and the diffusion classifier (Li et al. 2023) read the same network. Counting the network passes each makes per query, and weighting them by their measured cost per column, predicts which is cheaper:

with starts, iterations, field nodes, free coordinates (one input derivative each per node, for the Jacobian block), classes and noise draws; the trust region adds the energy at each trial point, Newton-or-descent a trial residual instead. pass_costs(pred, X, t, ps, st; directions) measures , and at the batch width the solvers use; root_cost, classifier_cost and break_even_classes evaluate the formulas.

Measured on MNIST (block model, image held clean, levels 0.05–0.2, 16 nodes, 1000 tuning images, RTX 3080): per column = 2.9 µs, = 1.1 µs, = 15.1 µs (a Hessian-vector product, forward over reverse, about five ε evaluations).

measuredpredictedaccuracy
root finding, Newton-or-descent (3.0 iterations)15.1 ms10.0 ms95.0%
root finding, trust region (3.8 iterations)18.5 ms11.9 ms95.0%
diffusion classifier, 10 × 32 draws3.1 ms2.0 ms93.9%
diffusion classifier, 10 × 12810.7 ms8.1 ms94.2%
diffusion classifier, 10 × 25621.5 ms16.2 ms93.7%

The predictions are 1.3 to 1.6 times too low: host-side work (assembling node inputs, transfers, narrower calls as queries leave the batch) is not counted. It is similar for both methods, so the ratio is right within 10% (5.0 predicted against 4.9 measured for Newton, 6.0 against 6.0 for the trust region, at 32 draws), and the ratio decides which to use.

What it says.

  • With one-hot labels, : root finding’s cost grows with the classes as fast as the classifier’s, with a larger constant ( against per class). It is never cheaper; here 5–6 times dearer at 32 draws.
  • With a label code of fixed dimension (an embedding instead of one-hot), root finding’s cost is fixed and the classifier’s grows with . At these units (the classifier called in wide batches, 2.9 µs per ε; in chunks of 100 as measured above it costs twice that) it is cheaper above classes (Newton-or-descent; with the trust region): about 45 classes for a 4-dimensional code, 110 for a 10-dimensional one.
  • The classifier’s accuracy did not grow with more draws (93.7–94.2% from 32 to 256), one point below root finding’s. Where accuracy matters, the comparison is not at equal accuracy at all.
  • The derivative passes dominate: is about 50 times here. Halving the nodes halves the cost; Newton-or-descent is cheaper than the trust region on easy queries; a cheaper Jacobian block (fewer directions by Krylov, or a reverse-mode Hessian) is where root finding gains most.

Related: analytic, reddiff, factor, energy_network, krylov, sampler, Deterministic Relaxation, Inference Signatures, The Implicit Diffusion Factor as a Statistical Game

Singular Jacobians in the pullback; non-conservative Laplace (2026-11-04, LP-023)

  • The dense implicit_pullback solve refuses a free-block Jacobian that is singular to working precision (, on a host copy; skipped for AD number types). At a fold or branch point the implicit function theorem does not apply, and the solve returned Inf/NaN. Tikhonov damping, which one review proposed, would return a gradient of a different problem silently. The Krylov path warns when GMRES stops short, as before.
  • implicit_laplace of a field that is not a gradient (a noise predictor that is not an energy network) returns the inverse of , which is then a local-curvature measure, not a posterior covariance; the docstring says so and a warning fires when .

Tests: test/review_singular.jl (a stub relation supplies the Jacobian).