Predictors and backends
Small networks, any AD backend
A diffusion model here is a prior over the joint space of one factor: a few coordinates, not an image. It is evaluated many times per query, so small models are the point, and the architecture is free. An MLP with a sinusoidal time embedding is the tested baseline; input adapts $(x, t)$ to whatever the model expects.
The implicit learner needs two derivatives of the network: the input Jacobian (Newton steps, adjoint solve) and a parameter VJP (backward pass). The backend is a field of the predictor:
using DifferentiationInterface, Zygote # or Enzyme, ForwardDiff, Mooncake, …
pred = NoisePredictor(mlp, VPSDE(); input, ad = AutoZygote())
using Reactant # XLA-compiled forward and Enzyme VJPs
pred = NoisePredictor(mlp, VPSDE(); input, ad = AutoReactant())Both are package extensions; without them the Jacobian falls back to finite differences. examples/circle_mlp.jl trains a 5k-parameter MLP on a circle in about 20 seconds on a CPU, then infers both branches and checks the adjoint against finite differences. For Enzyme on Lux use AutoEnzyme(; mode = Enzyme.set_runtime_activity(Enzyme.Reverse)).
Energy-parametrised models
NoisePredictor(EnergyNetwork(net), VPSDE(); input, ad = AutoZygote()): the network outputs a scalar energy and $\varepsilon_\theta = \sigma_t\nabla_x E_\theta$, so the score is conservative by construction. The implicit learner's field is then the gradient of implicit_energy, a real energy for the learned relation. Train with denoising_gradient (the loss gradient as one mixed second derivative, no nested AD in the loop) and any optimiser.
EnergyNetwork(net; reference = GaussianReference(X)) adds the exact energy of a diagonal Gaussian fitted to the data, $\sum_i (x_i - \alpha_i\mu_i)^2 / 2(\alpha_i^2 v_i + \sigma_i^2)$, and the network learns the residual. Its derivatives are added in closed form. At high noise, where a network's ε errors are amplified most in the denoised image, the reference already has the score up to the data's correlations. On MNIST it is what makes the samples of a convolutional energy clean.
On a GPU
device = (to = gpu_device(), back = cpu_device()) (MLDataDevices) runs the network and all its derivatives on the GPU, while the solvers keep their state on the CPU. Move the parameters yourself (ps = gpu_device()(ps)), and build the time features in input on the device of x. For an energy network use ad = SecondOrder(AutoForwardDiff(), AutoZygote()): the second derivatives are then taken forward over reverse on the device. At MNIST size (794 inputs, two hidden layers of 512) on an RTX 3080, a training step on 128 points takes 6 ms against 300 ms on the CPU. A classification query with ten free coordinates takes 1.8 ms per iteration when 1000 queries are solved together. Results agree with the CPU in Float64 to about 1e-7.
A noise level per block
NoisePredictor(net, VPSDE(); input, blocks = [1:784, 785:794]) lets blocks of coordinates sit at different noise levels ((Bao et al., 2023)). Time is then a number, one time per block, or a matrix with a column per input column, and input receives the matrix. Train with block_times(rng, pred, B) (independent times per block; clean holds one block at tmin in a share of the columns, as queries hold it) and denoising_gradient(…; weights) (a loss weight per coordinate, or per entry). A block held at a low level is a clean condition, a block at level 1 is noise: one network gives the joint score, every conditional and every marginal. The closed-form mixture and energy networks handle per-coordinate noise exactly, also on a device.
Closed-form and kernel predictors
GaussianMixtureEps is the exact noise predictor of a Gaussian mixture: every derivative the solvers need is in closed form, which makes it the test oracle. kde_predictor turns data points into such a mixture (a kernel density estimate), and OnlineKDE keeps one that learns from a stream and forgets.
API
The predictor
VariationalDiffusion.DeviceTransfer — Type
DeviceTransfer(to, back)Where a NoisePredictor runs: to moves arrays to the device, back brings results home. Built from the keyword device = (to = …, back = …).
VariationalDiffusion.NoiseBlocks — Type
NoiseBlocks(blocks)A partition of the coordinates 1:n into blocks that carry their own noise level, built from the keyword blocks of NoisePredictor (a vector of index ranges or vectors).
VariationalDiffusion.NoisePredictor — Type
NoisePredictor(model, schedule; input = default_input)$\varepsilon_\theta(x,t)$ — a wrapper making any Lux model into a diffusion prior.
model is any LuxCore.AbstractLuxLayer; parameters and state are delegated to it untouched, via AbstractLuxWrapperLayer, so ps for a NoisePredictor is ps for the model. Wrapping adds no parameters of its own.
input adapts the (x, t) pair to whatever the model expects, defaulting to the tuple (x, t). A model taking a concatenated time channel, a sinusoidal embedding or a NamedTuple is accommodated by passing a different input; the wrapper deliberately knows nothing about time embeddings, because that is the model's business.
pred = NoisePredictor(my_unet, VPSDE())
ps, st = LuxCore.setup(rng, pred) # == LuxCore.setup(rng, my_unet)
ε, st = epsilon(pred, x, 0.3, ps, st)device = (to = gpu_device(), back = cpu_device()) runs the network on an accelerator while the solvers keep their state on the CPU: arguments go to the device on every call, derivatives are taken there, results come back. The parameters must be moved to the device by the caller (ps = gpu_device()(ps)); their cotangents stay there. MLDataDevices' gpu_device() converts Float64 arrays to Float32, which a consumer GPU needs for speed. input then receives device arrays and must build any time embedding on the same device (predictor.md §5).
VariationalDiffusion.denoise — Method
denoise(p::NoisePredictor, x, t, ps, st) -> (x̂₀, st)Tweedie's formula: $\hat x_0 = (x - \sigma_t\varepsilon_\theta(x,t))/\alpha_t$, the posterior mean $\mathbb{E}[x_0 \mid x_t = x]$.
[!note] This is exact, not heuristic For a Gaussian data distribution the test suite checks
denoiseagainst the closed-form linear-Gaussian posterior mean and they agree to floating point. Tweedie's formula turns a noise predictor into an MMSE denoiser, and that is the sense in which a diffusion model is a prior: it is the object the RED-Diff regulariser scores against.
VariationalDiffusion.denoising_loss — Method
denoising_loss(p, x₀, t, ε, ps, st) -> (ℝ, st)$\|\varepsilon_\theta(\alpha_t x_0 + \sigma_t\varepsilon, t) - \varepsilon\|^2$, one Monte-Carlo sample of the score-matching objective at a single (t, ε).
Unweighted: the weighting $\omega(t)$ belongs to whoever is taking the expectation, and different weightings mean different things — the ELBO, the perceptual-quality objective, and RED-Diff's regulariser are the same integrand with three different $\omega$. See schedule.md §3.
VariationalDiffusion.epsilon — Method
epsilon(p::NoisePredictor, x, t, ps, st) -> (ε̂, st)One forward pass of the wrapped network: $\varepsilon_\theta(x, t)$.
This is the only place the network is evaluated forwards. RED-Diff's stop-gradient means sampling and RED-Diff need no reverse pass through model. Only the implicit learner's backward pass does (a parameter VJP and an input Jacobian), and it goes through the predictor's ad backend, provided by a package extension (predictor.md §5).
VariationalDiffusion.nblocks — Method
nblocks(p::NoisePredictor) -> IntThe number of noise blocks (1 without blocks).
VariationalDiffusion.noise_schedule — Method
noise_schedule(p::NoisePredictor) -> AbstractNoiseScheduleThe schedule the predictor was built with.
Named noise_schedule rather than schedule because the latter is Base.schedule (for Tasks), and shadowing it would be the sixth name collision in this project — see Mycelium.md §2 for the others.
VariationalDiffusion.score — Method
score(p::NoisePredictor, x, t, ps, st) -> (s, st)$s_\theta(x,t) = -\varepsilon_\theta(x,t)/\sigma_t \approx \nabla_x \log p_t(x)$.
The identity is exact for the perturbation kernel: with $x_t = \alpha_t x_0 + \sigma_t\varepsilon$,
\[\nabla_{x_t}\log p_{0t}(x_t\mid x_0) = -\frac{x_t - \alpha_t x_0}{\sigma_t^2} = -\frac{\varepsilon}{\sigma_t}\]
so a perfect noise predictor is a perfect score. The $1/\sigma_t$ is why VPSDE carries a tmin.
Closed-form predictors
VariationalDiffusion.GaussianMixtureEps — Type
GaussianMixtureEps(schedule, μ₀::AbstractMatrix; s = 0.05, weights = nothing)The exact noise predictor $\varepsilon^\ast$ of the data distribution $\sum_j w_j\,\mathcal N(\mu_j, s^2 I)$ under schedule, with equal weights $w_j = 1/J$ unless weights (non-negative, normalised internally) is given. Columns of μ₀ (an $n\times J$ matrix) are the initial component means; they are the layer's parameters (ps.μ), so a mixture can be moved by gradient descent. The component width s is fixed.
Wrap it in a NoisePredictor with the same schedule. Its input Jacobian (epsilon_jacobian) and parameter VJP (epsilon_vjp_params) are exact.
VariationalDiffusion.epsilon_jacobian — Method
epsilon_jacobian(pred::NoisePredictor, x, t, ps, st) -> Matrix$\partial\varepsilon_\theta/\partial x$ at $(x, t)$. The generic method uses central finite differences ($O(n)$ forward passes; fine for small $n$), or the predictor's AD backend when it has one (ad = …); closed-form predictors override it. For an exact score the matrix is symmetric — it is $-\sigma_t$ times a Hessian of $\log p_t$ — and the backward pass does not assume that, because a learned network's is not.
VariationalDiffusion.epsilon_jacobian_block — Method
epsilon_jacobian_block(pred::NoisePredictor, X, t, ps, st, idx, w) -> MatrixThe block $\sum_b w_b\,\partial\varepsilon_\theta(x_b, t)_{I}/\partial x_{b,I}$ of the input Jacobian on the coordinates $I$ = idx, summed over the columns $x_b$ of X (a vector is one column) with weights w. This is what the implicit solver needs: the Jacobian of the field on the free coordinates only, for all nodes of one noise level at once.
The cost grows with length(idx), not with the dimension: central differences along idx ($2|I|$ batched forward passes) without a backend; one AD Jacobian of a map $\mathbb R^{|I|}\to\mathbb R^{|I|}$ with one ($|I|$ JVPs or VJPs, whichever the backend does); closed form for GaussianMixtureEps; $|I|$ Hessian-vector products for an EnergyNetwork.
VariationalDiffusion.epsilon_jvp_input — Method
epsilon_jvp_input(pred::NoisePredictor, X, t, ps, st, V) -> ẊThe input Jacobian-vector product $\dot x_b = (\partial\varepsilon_\theta(x_b,t)/\partial x_b)\,V_b$, column by column, shaped like X. The matrix-free solver (KrylovSolve) needs one per noise level and Krylov step (prior_jvp). In forward mode through the predictor's AD backend when it has one; in closed form for GaussianMixtureEps and an EnergyNetwork; otherwise, and for reverse-only backends such as Zygote, by one central difference along V (two batched forward passes, accurate to about $10^{-8}$ relative).
VariationalDiffusion.epsilon_vjp_input — Method
epsilon_vjp_input(pred::NoisePredictor, X, t, ps, st, W) -> X̄The input vector–Jacobian product $\bar x_b = (\partial\varepsilon_\theta(x_b,t)/\partial x_b)^\top W_b$, column by column, shaped like X. The adjoint needs it once per noise level for the cotangent of the clamped inputs (prior_vjp). Through the predictor's AD backend when it has one, in closed form for GaussianMixtureEps, and from the finite-difference Jacobian otherwise.
VariationalDiffusion.epsilon_vjp_params — Method
epsilon_vjp_params(pred::NoisePredictor, x, t, ps, st, w) -> ps̄The vector–Jacobian product $w^\top\,\partial\varepsilon_\theta(x,t)/\partial\theta$, shaped like ps. This is the one place the backward pass needs a derivative with respect to the network's parameters.
- Closed-form predictors define it directly (e.g.
GaussianMixtureEps). - Any other model gets it from an AD backend chosen by the user: build the predictor with
NoisePredictor(model, schedule; ad = AutoZygote())(orAutoEnzyme(),AutoMooncake(),AutoForwardDiff(), … — anyADTypesbackend) and loadDifferentiationInterfacetogether with the backend package. This package itself depends on none of them.
Without either, it throws.
VariationalDiffusion.mixture_logdensity — Method
mixture_logdensity(l::GaussianMixtureEps, x, t, ps) -> Real$\log p_t(x)$ of the mixture. Used to check that the deterministic RED-Diff field is the gradient of a smoothed log-density (Implicit Diffusion Learners.md §3).
Kernel predictors
VariationalDiffusion.OnlineKDE — Type
OnlineKDE(dim; bandwidth, forget = 1.0, merge_radius = bandwidth / 2, budget = 1000,
prune = 1e-4)A Gaussian kernel density estimate that learns from a stream of samples and can forget. Feed it with observe!; query it through kde_predictor(schedule, kde), which returns a weighted GaussianMixtureEps, so every query function works on the current state.
Per observed sample:
- every existing weight is multiplied by
forget($\le 1$): a sample observed $k$ samples ago carries weight $\text{forget}^k$, so the effective memory is about $1/(1 - \text{forget})$ samples (forget = 1remembers everything, equally); - the sample is merged into the nearest centre if it lies within
merge_radius(the centre moves to the weighted mean, the weights add), and becomes a new centre otherwise; - centres whose weight fell below
prunetimes the largest are dropped (forgotten), and while more thanbudgetremain, the lightest centre is merged into its nearest neighbour (weighted mean, weights added), so compression never discards the newest samples.
Merging keeps the cost of a query bounded by the budget instead of growing with the stream; it changes the density by at most about merge_radius in each centre's position. The bandwidth is fixed; choose it on a first batch with kde_bandwidth.
VariationalDiffusion.kde_bandwidth — Method
kde_bandwidth(data; candidates = nothing, holdout = 0.2, rng = Xoshiro(0)) -> hThe bandwidth of an isotropic Gaussian KDE that maximises the held-out log-likelihood: a random fraction holdout of the columns of data is set aside, a KDE on the rest is evaluated on it, for each candidate. The default candidates span 0.5 % to 100 % of the data's scale (the root mean coordinate variance) on a log grid. This chooses the best density, not the best answers to queries, and uses no query information.
VariationalDiffusion.kde_predictor — Method
kde_predictor(schedule, data::AbstractMatrix; bandwidth) -> NoisePredictorA Gaussian kernel density estimate of the samples in the columns of data, as a noise predictor: GaussianMixtureEps with one component per sample and width bandwidth. Its $\varepsilon$ is the exact optimal noise predictor for that density, so every query function (implicit_infer, implicit_roots, implicit_laplace) works on it with exact derivatives and no training. It is the classical baseline for a diffusion network: the relation it defines is the ridge of the KDE. Query cost grows linearly with the number of samples. Choose the bandwidth with kde_bandwidth.
VariationalDiffusion.kde_predictor — Method
kde_predictor(schedule, kde::OnlineKDE) -> NoisePredictorThe current state of an OnlineKDE as a noise predictor: a weighted GaussianMixtureEps with the centres as parameters (ps.μ, from LuxCore.setup).
VariationalDiffusion.observe! — Method
observe!(kde::OnlineKDE, samples::AbstractMatrix) -> kde
observe!(kde::OnlineKDE, sample::AbstractVector) -> kdeAdd samples (columns) to the stream, in order: forget, merge or insert, prune. See OnlineKDE.
Energy networks
VariationalDiffusion.EnergyNetwork — Type
EnergyNetwork(model; reference = nothing)Marks model as a scalar energy $E_\theta(x, t)$ rather than a noise predictor. Wrapped in a NoisePredictor, it defines
\[\varepsilon_\theta(x, t) = \sigma_t\,\nabla_x E_\theta(x, t), \qquad s_\theta(x, t) = -\nabla_x E_\theta(x, t),\]
so the score is conservative by construction:
pred = NoisePredictor(EnergyNetwork(net), VPSDE(); input, ad = AutoZygote())net sees input(x, t) and returns one value per column of x (a 1 × B output, or a scalar for a single point). Parameters are the network's own. Every function of the package that takes a NoisePredictor accepts this one; the implicit learner additionally gains a scalar energy, implicit_energy.
Derivatives need an AD backend in ad, and second derivatives at that: the input Jacobian is a Hessian and the parameter VJP a mixed derivative. AutoZygote() and DifferentiationInterface.SecondOrder(AutoForwardDiff(), AutoZygote()) work with Lux layers (load DifferentiationInterface and the backend), as does SecondOrder(AutoMooncakeForward(), AutoMooncake()); Enzyme's forward-over-reverse on CPU does not yet, but ad = AutoReactant() (with Reactant loaded) runs it inside the compiled program (energy_network.md §4, §7).
With reference (a GaussianReference, a data matrix to fit one to, or a pair (μ, v)) the energy is the network's plus the exact energy of a diagonal Gaussian, $E_\theta + E_{\mathrm{ref}}$, and the network learns the residual. Its derivatives are added in closed form, without AD (energy_network.md §9).
VariationalDiffusion.GaussianReference — Type
GaussianReference(μ, v)
GaussianReference(X::AbstractMatrix; floor = 1e-4)A diagonal Gaussian $\mathcal N(\mu, \operatorname{diag} v)$ over the data, as the reference of an EnergyNetwork; from a data matrix (one sample per column) its mean and variance, the variance floored at floor. Noised to level $t$ it is $\mathcal N(\alpha_t\mu, \operatorname{diag}(\alpha_t^2 v + \sigma_t^2))$, so its energy is
\[E_{\mathrm{ref}}(x, t) = \sum_i \frac{(x_i - \alpha_i \mu_i)^2}{2(\alpha_i^2 v_i + \sigma_i^2)},\]
with $\alpha_i, \sigma_i$ of each coordinate's noise block. At high noise this is already the score up to the data's correlations, where a network's errors are amplified most when denoising; a coordinate that never varies ($v_i$ at the floor) is pulled to its mean at every level. The network adds what a diagonal Gaussian misses.
VariationalDiffusion.block_times — Method
block_times(rng, p::NoisePredictor, B; joint = 0.0, clean = 0.0) -> MatrixTraining times for a predictor with noise blocks: a (#blocks × B) matrix, each entry drawn uniformly from [tmin, 1] independently (UniDiffuser), so the network learns every block at every level: a block at tmin is a clean condition, a block at 1 is marginalised away. With joint the given fraction of the columns has one time shared by all blocks instead, the joint model's training distribution. With clean the given fraction of the remaining columns has one block, chosen uniformly, at exactly tmin: the condition of a query that holds that block clean (conditional_levels, clamp_level), which uniform times reach in only about a fraction 0.01 of the columns.
VariationalDiffusion.denoising_gradient — Method
denoising_gradient(p::NoisePredictor, x₀, t, ε, ps, st; weights = nothing) -> (loss, ps̄, st)The denoising loss $\frac1B\sum_i w_i(\hat\varepsilon_i - \varepsilon_i)^2$ over a batch (columns of x₀) and its gradient with respect to the parameters, through epsilon_vjp_params. t is a number, a 1 × B row, or with noise blocks a (#blocks × B) matrix (block_times): each coordinate is noised at its block's level, $x_t = \alpha \odot x_0 + \sigma \odot \varepsilon$. weights (one per coordinate, default all 1) weights the loss coordinate by coordinate: a small block such as a label needs a large weight to count next to a large one. A matrix the size of x₀ weights every entry, e.g. to leave out a block that a column holds clean.
For an EnergyNetwork this is the training step: the loss already contains a derivative of the network, and this computes the loss gradient as one mixed second derivative, with no nested AD in the training loop. Apply ps̄ with any optimiser.
VariationalDiffusion.energy — Method
energy(p::NoisePredictor{<:EnergyNetwork}, x, t, ps, st) -> (E, st)The network's energy $E_\theta(x, t)$: a scalar for a single point, the sum over columns for a batch. One forward pass; no derivative needed.
VariationalDiffusion.implicit_energy — Method
implicit_energy(m::ImplicitDiffusion, z, ps, st) -> (U, st)The scalar whose gradient is the implicit learner's field, for an energy-parametrised predictor:
\[U(z) = \sum_k w_k\,\lambda_{t_k}\Bigl[\tfrac{\sigma_k}{\alpha_k}\,E_\theta(\alpha_k z + \sigma_k\varepsilon_k,\ t_k) - \varepsilon_k^\top z\Bigr], \qquad \nabla U = g = \texttt{prior\_field}(m, z).\]
So the learned relation has an energy, and a query has a loss: $U(z) + \tfrac12\lVert P(z - z_0)\rVert^2$ on the free coordinates. For a noise predictor that outputs ε directly no such $U$ exists; see energy_network.md §3. A closed-form GaussianMixtureEps (and so a kernel density estimate) has one too, with $E = -\log p_t$ in closed form.