implementation

Energy-parametrised diffusion models: the network outputs a scalar energy and the noise prediction is its gradient, \varepsilon_\theta = \sigma_t\nabla_x E_\theta. The score is then conservative by construction, the implicit learner’s field is the exact gradient of a scalar, and the learned relation has a real energy. The cost is second derivatives, which the AD extension provides.

Sources: Salimans & Ho, Should EBMs model the energy or the score?, ICLR 2021 EBM workshop; Du et al., Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023; Bigi, Langer & Ceriotti, ICML 2025, arXiv:2412.11569 (the same distinction for force fields); code: energy_network.jl, ext/VariationalDiffusionDifferentiationInterfaceExt.jl

Bibliography: Salimans & Ho 2021 · Du et al. 2023 · Bigi et al. 2025

Theory: Geometric Deep Learning and Physical Laws · Implicit Diffusion Learners · The Implicit Diffusion Factor as a Statistical Game · backends

Notation. As in the rest of the vault: is the joint state; the network’s argument is written where it is the noised point .

1. Why: a direct score model is a direct force model

A noise predictor \varepsilon_\theta(x, t) is a learned vector field, and nothing makes it the gradient of anything. Its Jacobian is not symmetric (about 5% off for a trained circle model, backends §6), so the implicit learner’s field g is not the gradient of a scalar, the learned relation has no energy, and The Implicit Diffusion Factor as a Statistical Game §2 had to list “a scalar loss for learned networks” as missing.

This is the force-field situation of Geometric Deep Learning and Physical Laws §4: a model that predicts forces directly versus one that predicts an energy and differentiates. The tutorial Symmetry is not a law measures what the direct version costs there (circulation, energy drift, spurious resting points). EnergyNetwork is the energy version for diffusion.

2. The parametrisation

pred = NoisePredictor(EnergyNetwork(net), VPSDE(); input, ad = AutoZygote())

net returns one value per column of input(x, t). Then

The symmetry and scalar-field conclusions in §§1–5 assume a common noise time across coordinates of each point. With different block times, the noise Jacobian is and need not be symmetric; neither does the implicit field automatically have a scalar potential when its blocks interact.

EnergyNetwork is a marker wrapped inside NoisePredictor, the same pattern as NoisePredictor{<:GaussianMixtureEps}: epsilon, epsilon_jacobian and epsilon_vjp_params specialise on it, so ImplicitDiffusion, DiffusionFactor, RED-Diff and everything else accept an energy model unchanged.

3. What the implicit learner gains: an energy

With x_k = \alpha_k z + \sigma_k\varepsilon_k and \nabla_z E(x_k) = \alpha_k\nabla_x E(x_k),

implicit_energy(m, z, ps, st) returns U. Consequences:

  • A query has a loss, U(z) + \tfrac12\lVert P(z - z_0)\rVert^2. Two answers to the same query (the two branches of the circle) can be compared by it; a solver can do a line search on it.
  • stable means “local minimum” exactly: the field Jacobian is a Hessian, so its positive definiteness is the second-order condition, not a heuristic.
  • The adjoint uses J^\top = J, so the backward pass solves the same linear system as the forward Newton step.
  • The statistical-game reading is complete for learned networks: the prior game’s energy is U, not only its gradient (The Implicit Diffusion Factor as a Statistical Game §2).

4. Derivatives: one Hessian-vector product does the hard part

quantityneeded forcomputed as
\nabla_x E\varepsilon itselfDI.gradient with the first-order backend
free-coordinate noise JacobianNewton steps, adjoint solve, Laplaceone batched Hessian-vector product per free coordinate, scaled by each column’s row-wise noise coefficients before weighting and summing (implicit §6)
\nabla_x^2 E\, wcotangent of the clamped inputsDI.hvp
\partial_\theta\langle\nabla_x E, w\ranglebackward pass, trainingθ-block of DI.hvp on u = (x, \theta) in direction (w, 0)

The mixed derivative is the one that looks hard: a derivative with respect to the parameters of a derivative with respect to the input. Concatenating input and parameters into one vector turns it into an ordinary Hessian-vector product, which DifferentiationInterface computes forward-over-reverse.

Training needs no nested AD in the loop. The denoising loss \lVert\sigma_t\nabla_x E - \varepsilon\rVert^2 contains a derivative, so differentiating it naively is AD over AD. But its gradient is \partial_\theta\langle\nabla_x E, 2\sigma_t r\rangle with r the residual, the same mixed derivative. denoising_gradient(pred, x₀, t, ε, ps, st) returns the loss and that gradient for a batch; apply it with any optimiser.

Backends, checked on a two-layer tanh network against its closed-form Hessian (2026-10-12; Julia 1.12.6, Lux 1.31.4, LuxLib 1.15.9, Enzyme 0.13.216, Mooncake 0.5.63):

second-order backendresult
AutoZygote(), SecondOrder(AutoForwardDiff(), AutoZygote())exact to 1e-16; in the test suite
SecondOrder(AutoMooncakeForward(), AutoMooncake())exact to 3e-16; slow first call (about 70 s of compilation); not in the test suite
SecondOrder(AutoForwardDiff(), AutoMooncake())fails: Mooncake’s gradient rejects a dual-number return value
SecondOrder(AutoMooncake(), AutoMooncake())fails: Mooncake has no reverse-over-reverse
Enzyme forward-over-reverse inside Reactant.@compile (ad = AutoReactant())exact to 1e-16; in the Reactant suite (§7)
Enzyme forward-over-reverse on CPU (AutoEnzyme)fails, for two upstream reasons below

Enzyme on CPU fails twice over. LuxLib defines augmented_primal and reverse Enzyme rules for fused_dense! but no forward rule, so the outer forward pass through any Dense layer is a MethodError. Without Lux, sum(tanh.(W * x)) hits Enzyme.jl issue #3368 on Julia 1.12 (cannot handle unknown binary operator under runtime activity, which forward-over-reverse needs); Enzyme 0.13.181, from before that regression, runs it but returns a wrong Hessian. Neither affects Reactant, which differentiates the traced MLIR program, not the Julia code.

5. Validation

  • Exact. A Lux layer returning -\log p_t of a Gaussian mixture, wrapped as an EnergyNetwork, reproduces GaussianMixtureEps (whose \varepsilon is -\sigma_t\nabla\log p_t in closed form): \varepsilon, its Jacobian and the parameter VJP agree to 1e-16 with both backends.
  • Conservative. For an untrained Lux energy network the Jacobian is symmetric to 1e-17; \nabla U equals prior_field to 2e-13.
  • Backward pass. implicit_pullback through inference agrees with finite differences (relative 1e-8 for a weight), and denoising_gradient on a batch with a different t per column agrees with finite differences of the loss.
  • Training (§6).

6. Training on the circle, against an ε-network

The documentation’s tutorial A conservative score runs this comparison live. Same data (noisy points on the unit circle, t\in[0.001, 0.2]), same width (two hidden layers of 64), 6000 Adam steps at batch 256. The ε-network trains with Lux’s Training API; the energy network with denoising_gradient and Optimisers.

ε-networkenergy network
denoising loss per coordinate, last 500 steps≈ 0.540.547
stable answers to “y given x”, 38 queries3738
radius of the stable answers0.983 ± 0.0090.978 ± 0.008
asymmetry of the field Jacobian on the circle2.6%5·10⁻¹⁷
training time on a laptop CPU15 s41 s (SecondOrder(ForwardDiff, Zygote)), 52 s (AutoZygote)

The two fit equally well; the energy network costs about three times as much per step, the price of a second derivative. In return the field is a gradient to machine precision, and the relation has an energy that means something: along x = 0.6,

y+0.78 (upper branch)0-0.78 (lower branch)
U(0.6, y)−0.1850−0.1468−0.1839

Both branches sit at almost the same energy (the data are symmetric), with a barrier between them. A point method returns one branch; the energy says the other is equally good.

7. Implementation difficulties

  • The mixture code is not AD-friendly. _mixture_parts normalises the responsibilities in place (γ ./= sum(γ)), which Zygote cannot differentiate, so the exact test uses its own non-mutating mixture energy.
  • Float32 parameters pollute finite-difference checks. Lux initialises in Float32; a finite-difference check at step 10^{-5} against Float32 parameters disagrees at 1e-3. The tests convert parameters to Float64 first. On a GPU the AD derivatives themselves are fine in Float32 (§8): they agree with Float64 to about 1e-7.
  • The mixed derivative without a joint vector. The parameter VJP was the θ-block of one Hessian-vector product on , which flattens the parameters next to the input. With a forward outer backend (SecondOrder(AutoForwardDiff(), …)) it is now the derivative in of the parameter gradient at : . Only the input carries dual numbers, so the parameters can stay on a GPU. Other backends keep the joint form.
  • The input must accept a batch. input(x, t) is called with x a matrix and t a scalar or a 1\times B row during training, and with a vector during inference.
  • Reactant has its own five _energy_* hooks, since DifferentiationInterface has no Reactant backend. Each is Enzyme forward mode over a reverse gradient, compiled once per shape: the Hessian-vector product differentiates the input gradient, the mixed derivative the parameter gradient, both in the input direction. One program, the energy weighted per column , gives the gradient () and the free-coordinate block ( the field’s weights, one product per free coordinate). The Reactant suite checks them against finite differences of the plain forward energy, and inference, implicit_energy and the adjoint through them.

8. On a GPU

examples/gpu_energy.jl: an energy network at MNIST size, on , 802 → 512 → 512 → 1. It uses ad = SecondOrder(AutoForwardDiff(), AutoZygote()) and device = (to = gpu_device(), back = cpu_device()), so the network runs in Float32 on an RTX 3080 while the solver’s state stays in Float64 on the CPU. The reference is the same network on the CPU in Float64.

GPUCPU (8 threads)agreement
training step, 128 points (loss and parameter gradient)6.4 ms299 msgradient 3e-8
ε for 4000 columns10 ms4e-7
Hessian-vector product, 4000 columns21 ms9e-7
Hessian block of 400 columns6e-8
classification-shaped solves (784 clamped, 10 free, 16 nodes), per query and iteration1.8 ms at 1000 queries, 2.6 ms at 100, 24 ms alone324 ms (100 queries)iterates 2e-12

So training an energy network on MNIST is ordinary GPU training at a few milliseconds per batch. Classifying the 10,000 test images from all ten class codes (about six iterations each) is about 20 minutes. Batched queries (implicit §7) are what make this work: one query alone spends its time on transfers and launches. Without the GPU, Lux’s nested AD (StatefulLuxLayer, Zygote over Zygote) failed on this network, on the CPU and on the GPU. The package does not need it, because the mixed derivative is taken forward over reverse (§7).

9. A Gaussian reference energy

EnergyNetwork(net; reference = GaussianReference(X)) adds the exact energy of a diagonal Gaussian fitted to the data (mean , variance floored at ):

with of the coordinate’s noise block. The network learns the residual. Its derivatives are added in closed form, outside the AD backend: to the gradient, to the Hessian, to a Hessian-vector product, and to the diagonal of the free-coordinate noise Jacobian block. It has no parameters, so the mixed derivative of training is unchanged.

Why it helps (MNIST, MNIST in Both Directions §9): at high noise the denoiser multiplies the network’s ε error by , up to 150 at , and a pixel that is background in every image needs a score that the network learns poorly. The reference gives the right score at high noise up to the data’s correlations, and pulls a never-varying pixel to its mean with curvature . With it, a convolutional energy samples clean digits (border speckles 1.7% from , against 18.6% without).

The test checks every hook against the same reference written into the network and differentiated by AD, with and without noise blocks, on the CPU and on a device in Float64 and Float32. On the MNIST model the closed-form version reproduces the AD version’s denoising errors and sampling statistics to the printed digits. With the derivative of the reference no longer in the AD graph, the classification queries of the convolutional energy fit in chunks of 100 on a 10 GB GPU instead of 50.

Related: predictor, implicit, backends, analytic, Geometric Deep Learning and Physical Laws, The Implicit Diffusion Factor as a Statistical Game

LP-018 per-column Jacobian scales (2026-10-29)

The free-coordinate Jacobian is , including the reference’s diagonal Hessian. Each basis direction now uses a batched Hessian-vector product, scales each column before summing, and restricts to the requested rows. Factoring out the first column’s scale was only valid for common times. The single-point Jacobian, input JVP/VJP and mixture-specific methods were checked as well; the mixture methods already use each column’s times. With different row scales, the noise Jacobian need not be symmetric even for an energy. Evidence: test/lp018.jl, “LP-018.3 energy block Jacobians weight each column”, against explicit quadratic Hessians, with cross terms, reference energies and reordered/subset coordinates.