VariationalDiffusion

A diffusion model as a factor: wrap a trained noise predictor and use it as a prior over the joint space of a factor, queried by root finding (the implicit learner), by RED-Diff's proximal solve, or by ProxDM's learned prox.

This package consumes a trained $\varepsilon_\theta$ (it offers training losses, not a training loop) and has no automatic-differentiation dependency: derivatives come from closed forms, finite differences, or an AD backend the user chooses through a package extension.

pagecovers
this pagethe pieces, the schedule, choosing λ, known gaps
Predictors and backendsNoisePredictor, closed-form and kernel predictors, AD and Reactant backends, energy networks
Implicit inferencethe solver, all answers and their uncertainty, many free coordinates, sampling, gradients through inference, products
FactorsDiffusionFactor with RED-Diff or the implicit solver as its inversion
ProxDMproximal diffusion models: the prox, the sampler, proximal inference

The pieces

sched = VPSDE()                                   # the forward process
pred  = NoisePredictor(my_unet, sched)            # ε_θ(x, t) — any Lux model
prox  = REDDiff(λ = 0.25, steps = 200)            # the inversion
f     = DiffusionFactor((obs = 4, hidden = 4), pred; prox)

The schedule

VPSDE is the variance-preserving SDE of Song et al.: $x_t = \alpha_t x_0 + \sigma_t \varepsilon$ with $\alpha_t^2 + \sigma_t^2 = 1$. Query it with alpha, sigma, snr, perturb, sample_time.

The predictor

Details, backends and energy networks: Predictors and backends.

NoisePredictor wraps any LuxCore.AbstractLuxLayer and derives two things from it:

functionisnote
epsilonthe network itselfthe only place it is evaluated
score$-\varepsilon_\theta/\sigma_t$$\approx \nabla_x \log p_t(x)$
denoise$(x - \sigma_t\varepsilon_\theta)/\alpha_t$Tweedie — the MMSE denoiser

Parameters are the wrapped model's, untouched: LuxCore.setup(rng, pred) == LuxCore.setup(rng, unet).

The factor

Details: Factors.

DiffusionFactor carves one vector space into named channel blocks. The Polarity decides which blocks are clamped and how hard, which is exactly the selection-matrix formulation $P = \rho_{in}P_{in} + \rho_{out}P_{out} + \rho_{latent}P_{latent}$ — inpainting, in other words. Observed() defaults to $\rho = \infty$, a hard clamp applied by projection.

The inversion runs REDDiff and returns a DiracBelief, because RED-Diff's variational family is a point mass. That is the paper's choice, not a simplification here.

Choosing λ

λ is presented in the paper as a tuned hyperparameter (they use 0.25). For a Gaussian data distribution it is derivable — there is exactly one value making the implied prior correct, and calibrate_lambda(schedule, v₀) returns it.

Worth knowing what tuning λ actually does: it sets how strong the learned prior is. A single scalar can only calibrate one eigendirection of a correlated prior, so it over-regularises high-variance directions and under-regularises low-variance ones.

Known gaps

  • A ProxDM factor sends point messages only; there is no Laplace Gaussian for proximal inference yet.
  • RED-Diff's inversion returns a point, so uncertainty does not propagate; the implicit solver's message = :gaussian does send a Laplace Gaussian.
  • implicit_sample takes hard clamps only; soft evidence would need likelihood guidance.
  • A separator of a product is a relation the user supplies (trained on the shared marginal); it is not derived from the factors.
  • The KrylovSolve stability test and log-determinant are estimates once more coordinates are free than Lanczos steps and probes.

Index

API

VariationalDiffusion.VariationalDiffusion — Module
VariationalDiffusion

A diffusion model as a statistical game: the third of the three Implicit Learners.md families, and the only one whose Bayesian inversion is neither exact nor a root-find but a proximal solve.

The chain

  1. schedule.jl — the VP-SDE of Song et al. 2021 (arXiv:2011.13456), presented through its perturbation kernel $p_{0t}(x_t\mid x_0) = \mathcal{N}(\alpha_t x_0, \sigma_t^2 I)$ with $\alpha_t^2 + \sigma_t^2 = 1$.

  2. predictor.jl — NoisePredictor wraps any Lux model as $\varepsilon_\theta(x,t)$ and derives from it the score $-\varepsilon_\theta/\sigma_t$ and Tweedie's denoiser $(x - \sigma_t\varepsilon_\theta)/\alpha_t$.

  3. reddiff.jl — REDDiff, the proximal operator of Mardani et al. 2023 (arXiv:2305.04391): variational inference with a point-mass posterior, whose regulariser gradient is $\mathbb{E}_{t,\varepsilon}[\lambda_t(\varepsilon_\theta(x_t,t)-\varepsilon)]$ under a stop-gradient.

  4. factor.jl — DiffusionFactor, where ImplicitREDDiff.md's selection matrices $P_{in} + P_{out} + P_{latent} = \mathrm{Id}$ become a LenticulumCore.Polarity and

    \[E(x_0, x) = \underbrace{\mathbb{E}_{t,\varepsilon}\bigl[\omega(t)\|\varepsilon_\theta(\alpha_t x + \sigma_t\varepsilon,t)-\varepsilon\|^2\bigr]}_{\text{denoising surrogate}} \;+\; \underbrace{\tfrac12\|P(x_0-x)\|^2}_{\text{energy } \mathbf{l}^c}\]

Two things worth knowing before reading the code

No automatic differentiation anywhere. RED-Diff's stop-gradient means the denoiser's Jacobian is never formed, and the clamp's gradient is $P^2(x-x_0)$ in closed form. So the entire inversion is forward passes plus arithmetic, and this package's dependencies are LuxCore, Random and LinearAlgebra. Lux itself is not a dependency — any Lux model is an AbstractLuxLayer, which is all the wrapper needs.

The inversion returns a DiracBelief, because RED-Diff's variational family is $\mathcal{N}(\mu,\sigma^2 I)$ with $\sigma\to 0$. That is the paper's own choice, not a simplification made here, and it has consequences on a graph — see factor.md §5.

Concept notes are in vault/Families/Diffusion/; per-file implementation notes sit next to each source file, per vault/Start Here.md.

source
VariationalDiffusion.AbstractNoiseSchedule — Type
abstract type AbstractNoiseSchedule

A forward corruption process, presented through its perturbation kernel rather than its SDE: everything downstream needs only alpha(s, t) and sigma(s, t).

That is a deliberate narrowing. The SDE is the definition; the Gaussian marginal is what RED-Diff and score matching actually touch, and a schedule whose marginal is not Gaussian would not fit this interface. drift and diffusion are provided for completeness and are not used by the inversion.

source
VariationalDiffusion.VPSDE — Type
VPSDE(; βmin = 0.1, βmax = 20.0, tmin = 1e-3)

The variance-preserving SDE. Defaults are Song et al.'s, which are in turn DDPM's discretisation in the continuous limit.

tmin is a sampling floor, not part of the mathematics: at t = 0 we have σ_t = 0, the score -ε_θ/σ_t is a division by zero, and the RED-Diff weight λ_t = λσ_t/α_t degenerates. Every published implementation carries such a floor and most do not say so. Times are drawn from [tmin, 1]; see sample_time.

source
VariationalDiffusion.drift — Method
drift(s::VPSDE, x, t)

The SDE's drift coefficient $f(x,t) = -\tfrac12\beta(t)x$; see diffusion for $g(t)$.

Provided for completeness and to document what the schedule is. Nothing in this package integrates the SDE — RED-Diff replaces sampling with optimisation, which is the whole point (RED-Diff as a Statistical Game.md §2).

source
VariationalDiffusion.integrated_beta — Method
integrated_beta(s::VPSDE, t) -> Real

$B(t) = \int_0^t \beta(u)\,du = \beta_{\min}t + \tfrac12(\beta_{\max}-\beta_{\min})t^2$.

In closed form, which is the only reason the VP-SDE is convenient: a schedule needing numerical quadrature here would put a solve inside every message.

source
VariationalDiffusion.marginal_variance — Method
marginal_variance(s, t, v₀) -> Real

The variance of $x_t$ when $x_0$ has variance v₀: $\alpha_t^2 v_0 + \sigma_t^2$.

For v₀ = 1 this is identically 1 — the variance-preserving property. It is used by the analytic Gaussian oracle in the test suite, which is the only closed-form $\varepsilon_\theta$ available and therefore the only thing the inversion can be checked against.

source
VariationalDiffusion.perturb — Method
perturb(s, x₀, t, ε) -> x_t

$x_t = \alpha_t x_0 + \sigma_t \varepsilon$: one draw from the perturbation kernel, written as a reparametrisation so it is differentiable in x₀.

That differentiability is exactly what RED-Diff keeps and what its stop-gradient does not discard — see reddiff.md §2.

source
VariationalDiffusion.sigma — Method
sigma(s, t) -> Real

The noise coefficient $\sigma_t = \sqrt{1 - \exp(-B(t))} = \sqrt{1 - \alpha_t^2}$.

Computed from expm1 rather than as sqrt(1 - alpha^2): near t = 0 the latter is sqrt(1 - (1-ε)²) and loses half its significant digits to cancellation.

source
VariationalDiffusion.snr — Method
snr(s, t) -> Real

The amplitude ratio $\alpha_t/\sigma_t = \sqrt{\mathrm{SNR}_t}$, where the power SNR is $\alpha_t^2/\sigma_t^2$. RED-Diff's weighting is $\lambda_t = \lambda/\sqrt{\mathrm{SNR}_t}$; see REDDiff.

source