VariationalDiffusion
A diffusion model as a factor: wrap a trained noise predictor and use it as a prior over the joint space of a factor, queried by root finding (the implicit learner), by RED-Diff's proximal solve, or by ProxDM's learned prox.
This package consumes a trained $\varepsilon_\theta$ (it offers training losses, not a training loop) and has no automatic-differentiation dependency: derivatives come from closed forms, finite differences, or an AD backend the user chooses through a package extension.
| page | covers |
|---|---|
| this page | the pieces, the schedule, choosing λ, known gaps |
| Predictors and backends | NoisePredictor, closed-form and kernel predictors, AD and Reactant backends, energy networks |
| Implicit inference | the solver, all answers and their uncertainty, many free coordinates, sampling, gradients through inference, products |
| Factors | DiffusionFactor with RED-Diff or the implicit solver as its inversion |
| ProxDM | proximal diffusion models: the prox, the sampler, proximal inference |
The pieces
sched = VPSDE() # the forward process
pred = NoisePredictor(my_unet, sched) # ε_θ(x, t) — any Lux model
prox = REDDiff(λ = 0.25, steps = 200) # the inversion
f = DiffusionFactor((obs = 4, hidden = 4), pred; prox)The schedule
VPSDE is the variance-preserving SDE of Song et al.: $x_t = \alpha_t x_0 + \sigma_t \varepsilon$ with $\alpha_t^2 + \sigma_t^2 = 1$. Query it with alpha, sigma, snr, perturb, sample_time.
The predictor
Details, backends and energy networks: Predictors and backends.
NoisePredictor wraps any LuxCore.AbstractLuxLayer and derives two things from it:
| function | is | note |
|---|---|---|
epsilon | the network itself | the only place it is evaluated |
score | $-\varepsilon_\theta/\sigma_t$ | $\approx \nabla_x \log p_t(x)$ |
denoise | $(x - \sigma_t\varepsilon_\theta)/\alpha_t$ | Tweedie — the MMSE denoiser |
Parameters are the wrapped model's, untouched: LuxCore.setup(rng, pred) == LuxCore.setup(rng, unet).
The factor
Details: Factors.
DiffusionFactor carves one vector space into named channel blocks. The Polarity decides which blocks are clamped and how hard, which is exactly the selection-matrix formulation $P = \rho_{in}P_{in} + \rho_{out}P_{out} + \rho_{latent}P_{latent}$ — inpainting, in other words. Observed() defaults to $\rho = \infty$, a hard clamp applied by projection.
The inversion runs REDDiff and returns a DiracBelief, because RED-Diff's variational family is a point mass. That is the paper's choice, not a simplification here.
Choosing λ
λ is presented in the paper as a tuned hyperparameter (they use 0.25). For a Gaussian data distribution it is derivable — there is exactly one value making the implied prior correct, and calibrate_lambda(schedule, v₀) returns it.
Worth knowing what tuning λ actually does: it sets how strong the learned prior is. A single scalar can only calibrate one eigendirection of a correlated prior, so it over-regularises high-variance directions and under-regularises low-variance ones.
Known gaps
- A
ProxDMfactor sends point messages only; there is no Laplace Gaussian for proximal inference yet. - RED-Diff's inversion returns a point, so uncertainty does not propagate; the implicit solver's
message = :gaussiandoes send a Laplace Gaussian. implicit_sampletakes hard clamps only; soft evidence would need likelihood guidance.- A separator of a product is a relation the user supplies (trained on the shared marginal); it is not derived from the factors.
- The
KrylovSolvestability test and log-determinant are estimates once more coordinates are free than Lanczos steps and probes.
Index
VariationalDiffusion.VariationalDiffusionVariationalDiffusion.AbstractImplicitRelationVariationalDiffusion.AbstractLinearSolveVariationalDiffusion.AbstractNoiseScheduleVariationalDiffusion.AbstractProximalPredictorVariationalDiffusion.CoverageReferenceVariationalDiffusion.DenseSolveVariationalDiffusion.DeviceTransferVariationalDiffusion.DiffusionFactorVariationalDiffusion.DiffusionModelVariationalDiffusion.EnergyNetworkVariationalDiffusion.FieldNodesVariationalDiffusion.GaussianMixtureEpsVariationalDiffusion.GaussianReferenceVariationalDiffusion.ImplicitDiffusionVariationalDiffusion.ImplicitProxVariationalDiffusion.ImplicitSolutionVariationalDiffusion.KrylovSolveVariationalDiffusion.MixtureProxVariationalDiffusion.NoiseBlocksVariationalDiffusion.NoisePredictorVariationalDiffusion.OnlineKDEVariationalDiffusion.ProductRelationVariationalDiffusion.ProxDMVariationalDiffusion.ProxNetworkVariationalDiffusion.QueryReportVariationalDiffusion.REDDiffVariationalDiffusion.VPSDEVariationalDiffusion.alphaVariationalDiffusion.assemble_stateVariationalDiffusion.betaVariationalDiffusion.block_timesVariationalDiffusion.blockrangesVariationalDiffusion.break_even_classesVariationalDiffusion.calibrate_lambdaVariationalDiffusion.calibrate_temperatureVariationalDiffusion.categorical_answerVariationalDiffusion.classifier_costVariationalDiffusion.code_posteriorVariationalDiffusion.conditional_levelsVariationalDiffusion.coverageVariationalDiffusion.coverage_referenceVariationalDiffusion.default_toleranceVariationalDiffusion.denoiseVariationalDiffusion.denoising_gradientVariationalDiffusion.denoising_lossVariationalDiffusion.density_lambdaVariationalDiffusion.diffusionVariationalDiffusion.driftVariationalDiffusion.energyVariationalDiffusion.epsilonVariationalDiffusion.epsilon_jacobianVariationalDiffusion.epsilon_jacobian_blockVariationalDiffusion.epsilon_jvp_inputVariationalDiffusion.epsilon_vjp_inputVariationalDiffusion.epsilon_vjp_paramsVariationalDiffusion.field_nodesVariationalDiffusion.implicit_classifyVariationalDiffusion.implicit_classifyVariationalDiffusion.implicit_energyVariationalDiffusion.implicit_factor_pullbackVariationalDiffusion.implicit_inferVariationalDiffusion.implicit_infer_manyVariationalDiffusion.implicit_laplaceVariationalDiffusion.implicit_mixtureVariationalDiffusion.implicit_pullbackVariationalDiffusion.implicit_residualVariationalDiffusion.implicit_rootsVariationalDiffusion.implicit_sampleVariationalDiffusion.implicit_solutionVariationalDiffusion.integrated_betaVariationalDiffusion.kde_bandwidthVariationalDiffusion.kde_predictorVariationalDiffusion.kde_predictorVariationalDiffusion.laplace_beliefVariationalDiffusion.marginal_varianceVariationalDiffusion.mixture_logdensityVariationalDiffusion.nblocksVariationalDiffusion.noise_scheduleVariationalDiffusion.noisefree_nodesVariationalDiffusion.observe!VariationalDiffusion.pass_costsVariationalDiffusion.perturbVariationalDiffusion.precision_vectorVariationalDiffusion.prior_fieldVariationalDiffusion.prior_jacobianVariationalDiffusion.prior_jvpVariationalDiffusion.prior_vjpVariationalDiffusion.prox_inferVariationalDiffusion.prox_jacobianVariationalDiffusion.prox_pullbackVariationalDiffusion.prox_vjp_inputVariationalDiffusion.prox_vjp_paramsVariationalDiffusion.proxdm_sampleVariationalDiffusion.proximalVariationalDiffusion.proximal_matching_lossVariationalDiffusion.query_reportVariationalDiffusion.reddiff_solveVariationalDiffusion.reddiff_weightVariationalDiffusion.regulariser_gradientVariationalDiffusion.ridge_residualVariationalDiffusion.root_costVariationalDiffusion.round_tripVariationalDiffusion.sample_timeVariationalDiffusion.scoreVariationalDiffusion.sigmaVariationalDiffusion.snrVariationalDiffusion.statedimVariationalDiffusion.trusted
API
VariationalDiffusion.VariationalDiffusion — Module
VariationalDiffusionA diffusion model as a statistical game: the third of the three Implicit Learners.md families, and the only one whose Bayesian inversion is neither exact nor a root-find but a proximal solve.
The chain
schedule.jl— the VP-SDE of Song et al. 2021 (arXiv:2011.13456), presented through its perturbation kernel $p_{0t}(x_t\mid x_0) = \mathcal{N}(\alpha_t x_0, \sigma_t^2 I)$ with $\alpha_t^2 + \sigma_t^2 = 1$.predictor.jl—NoisePredictorwraps any Lux model as $\varepsilon_\theta(x,t)$ and derives from it the score $-\varepsilon_\theta/\sigma_t$ and Tweedie's denoiser $(x - \sigma_t\varepsilon_\theta)/\alpha_t$.reddiff.jl—REDDiff, the proximal operator of Mardani et al. 2023 (arXiv:2305.04391): variational inference with a point-mass posterior, whose regulariser gradient is $\mathbb{E}_{t,\varepsilon}[\lambda_t(\varepsilon_\theta(x_t,t)-\varepsilon)]$ under a stop-gradient.factor.jl—DiffusionFactor, whereImplicitREDDiff.md's selection matrices $P_{in} + P_{out} + P_{latent} = \mathrm{Id}$ become aLenticulumCore.Polarityand\[E(x_0, x) = \underbrace{\mathbb{E}_{t,\varepsilon}\bigl[\omega(t)\|\varepsilon_\theta(\alpha_t x + \sigma_t\varepsilon,t)-\varepsilon\|^2\bigr]}_{\text{denoising surrogate}} \;+\; \underbrace{\tfrac12\|P(x_0-x)\|^2}_{\text{energy } \mathbf{l}^c}\]
Two things worth knowing before reading the code
No automatic differentiation anywhere. RED-Diff's stop-gradient means the denoiser's Jacobian is never formed, and the clamp's gradient is $P^2(x-x_0)$ in closed form. So the entire inversion is forward passes plus arithmetic, and this package's dependencies are LuxCore, Random and LinearAlgebra. Lux itself is not a dependency — any Lux model is an AbstractLuxLayer, which is all the wrapper needs.
The inversion returns a DiracBelief, because RED-Diff's variational family is $\mathcal{N}(\mu,\sigma^2 I)$ with $\sigma\to 0$. That is the paper's own choice, not a simplification made here, and it has consequences on a graph — see factor.md §5.
Concept notes are in vault/Families/Diffusion/; per-file implementation notes sit next to each source file, per vault/Start Here.md.
VariationalDiffusion.AbstractNoiseSchedule — Type
abstract type AbstractNoiseScheduleA forward corruption process, presented through its perturbation kernel rather than its SDE: everything downstream needs only alpha(s, t) and sigma(s, t).
That is a deliberate narrowing. The SDE is the definition; the Gaussian marginal is what RED-Diff and score matching actually touch, and a schedule whose marginal is not Gaussian would not fit this interface. drift and diffusion are provided for completeness and are not used by the inversion.
VariationalDiffusion.VPSDE — Type
VPSDE(; βmin = 0.1, βmax = 20.0, tmin = 1e-3)The variance-preserving SDE. Defaults are Song et al.'s, which are in turn DDPM's discretisation in the continuous limit.
tmin is a sampling floor, not part of the mathematics: at t = 0 we have σ_t = 0, the score -ε_θ/σ_t is a division by zero, and the RED-Diff weight λ_t = λσ_t/α_t degenerates. Every published implementation carries such a floor and most do not say so. Times are drawn from [tmin, 1]; see sample_time.
VariationalDiffusion.alpha — Method
alpha(s, t) -> RealThe signal coefficient $\alpha_t = \exp(-\tfrac12 B(t))$ of the perturbation kernel.
VariationalDiffusion.beta — Method
beta(s::VPSDE, t) -> Real$\beta(t) = \beta_{\min} + t(\beta_{\max} - \beta_{\min})$, the SDE's noise rate.
VariationalDiffusion.diffusion — Method
diffusion(s::VPSDE, t) -> RealThe SDE's diffusion coefficient $g(t) = \sqrt{\beta(t)}$. See drift.
VariationalDiffusion.drift — Method
drift(s::VPSDE, x, t)The SDE's drift coefficient $f(x,t) = -\tfrac12\beta(t)x$; see diffusion for $g(t)$.
Provided for completeness and to document what the schedule is. Nothing in this package integrates the SDE — RED-Diff replaces sampling with optimisation, which is the whole point (RED-Diff as a Statistical Game.md §2).
VariationalDiffusion.integrated_beta — Method
integrated_beta(s::VPSDE, t) -> Real$B(t) = \int_0^t \beta(u)\,du = \beta_{\min}t + \tfrac12(\beta_{\max}-\beta_{\min})t^2$.
In closed form, which is the only reason the VP-SDE is convenient: a schedule needing numerical quadrature here would put a solve inside every message.
VariationalDiffusion.marginal_variance — Method
marginal_variance(s, t, v₀) -> RealThe variance of $x_t$ when $x_0$ has variance v₀: $\alpha_t^2 v_0 + \sigma_t^2$.
For v₀ = 1 this is identically 1 — the variance-preserving property. It is used by the analytic Gaussian oracle in the test suite, which is the only closed-form $\varepsilon_\theta$ available and therefore the only thing the inversion can be checked against.
VariationalDiffusion.perturb — Method
perturb(s, x₀, t, ε) -> x_t$x_t = \alpha_t x_0 + \sigma_t \varepsilon$: one draw from the perturbation kernel, written as a reparametrisation so it is differentiable in x₀.
That differentiability is exactly what RED-Diff keeps and what its stop-gradient does not discard — see reddiff.md §2.
VariationalDiffusion.sample_time — Method
sample_time(rng, s) -> RealA time drawn uniformly from [tmin, 1], the outer expectation of the score-matching regulariser.
VariationalDiffusion.sigma — Method
sigma(s, t) -> RealThe noise coefficient $\sigma_t = \sqrt{1 - \exp(-B(t))} = \sqrt{1 - \alpha_t^2}$.
Computed from expm1 rather than as sqrt(1 - alpha^2): near t = 0 the latter is sqrt(1 - (1-ε)²) and loses half its significant digits to cancellation.
VariationalDiffusion.snr — Method
snr(s, t) -> RealThe amplitude ratio $\alpha_t/\sigma_t = \sqrt{\mathrm{SNR}_t}$, where the power SNR is $\alpha_t^2/\sigma_t^2$. RED-Diff's weighting is $\lambda_t = \lambda/\sqrt{\mathrm{SNR}_t}$; see REDDiff.