Implicit inference

The implicit learner turns a noise predictor into a relation: a fixed set of field nodes makes RED-Diff's regulariser gradient a deterministic field $g_\theta$, and a query clamps some coordinates (ρ = Inf), anchors others softly (finite ρ) and solves $g_\theta(z) + \rho^2 \odot (z - z_0) = 0$ on the rest.

nodes = field_nodes(rng, n)                       # fixed (t, ε, w)
m     = ImplicitDiffusion(pred, nodes; batched = true)
sol, st = implicit_infer(m, z₀, ρ, ps, st)        # a root, with converged / stable flags
b = implicit_pullback(m, sol, z₀, ρ, z̄, ps, st)   # gradients by the adjoint

All answers, and how sure

implicit_roots solves from several starts and keeps the distinct stable answers; implicit_laplace and laplace_belief give the Laplace covariance at an answer; implicit_mixture combines all answers into a MixtureBelief whose weights are Laplace mass estimates when the model has an energy. Build the model with λ = density_lambda(schedule, nodes) for covariances in data units. round_trip checks an answer against the relation: it frees the clamped coordinates, clamps the rest at the answer and solves back; an answer on the relation stays, a projection of a query off the relation moves its clamps. ridge_residual checks it from the field at the answer alone: the field's component across the relation (the eigenvectors of the Jacobian's largest curvatures, as many as the codimension), divided by those curvatures, is the first-order distance to the ridge in data units. Unlike the round trip it does not reject an answer where the relation runs parallel to a clamped coordinate.

Many free coordinates

Every solver step, the Laplace covariance and the adjoint need only the Jacobian block on the free coordinates, and that is all they compute. With ImplicitDiffusion(pred, nodes; batched = true) all nodes of one noise level go through the network in one call. When most of the state is free (generating an image), pass linsolve = KrylovSolve() to implicit_infer, implicit_pullback, laplace_belief, implicit_mixture or ImplicitProx: Newton steps and the adjoint are then solved by GMRES on Jacobian-vector products (prior_jvp) and vector-Jacobian products (prior_vjp), stability is tested by Lanczos, and the Laplace covariance is computed only on the coordinates asked for. On a 794-dimensional MLP with 784 free coordinates, implicit_infer takes about 1 s; one dense 784 × 784 block takes 33 s.

Many queries, and classification

implicit_infer_many(m, Z₀, ρ, ps, st) solves the columns of Z₀ as separate queries with one polarity, and evaluates all of them in each network call: one call per noise level for the field, one more for the free-coordinate blocks. The answers are those of implicit_infer query by query. implicit_classify(m, z₀, ρ, ps, st; label, classes) frees the label coordinates, starts from every class code and returns a CategoricalBelief. The mass of roots far from every code is reported as unassigned, and the Gaussian mixture the belief was read from is kept. Given a matrix of queries, it solves them all at once. On three classes with a closed-form model the class probabilities are within 0.04 of the exact posterior.

With a predictor with noise blocks, ImplicitDiffusion(pred, nodes; levels = conditional_levels(pred, ρ)) holds the clamped blocks clean at every node, so the field on the free coordinates is the smoothed conditional score. On the unit circle this moves the answers from radius 0.978 to 0.993 (field levels up to 0.05). When a query has a single root, categorical_answer(…; temperature, cov_scale) reads class probabilities from the root's position and covariance, with both fitted on held-out answers by calibrate_temperature (on MNIST a calibration error of 0.009). The solvers' default tolerance follows the arithmetic precision (default_tolerance). For relations with an energy, implicit_infer uses a trust-region method on the query's energy (method = :auto; (Nocedal and Wright, 2006), (Steihaug, 1983)): on the circle every one of 160 starts converges (against 122 for Newton-or-descent), and on MNIST 19 of 20 modes with 784 free pixels, in about 6 s each, where Newton-or-descent found none. implicit_infer_many uses the same method for many queries at once. pass_costs, root_cost and classifier_cost predict the time per query of root finding and of the diffusion classifier ((Li et al., 2023)) from measured per-column costs; on MNIST the ratio is right within 10%, and with a compact label code break_even_classes gives the number of classes above which root finding is cheaper.

Off the data

Inference answers every query, also one the data never covered. coverage_reference(m, data, ps, st; calibration) scores held-out points; coverage(ref, sol) then ranks an answer by its distance to the data, the denoiser's reconstruction error and, with an energy, $-\log p_t$, each as a conformal p-value ((Bates et al., 2023)). On the unit circle about 5% of fresh data points get p < 0.05, and the answer to the clamp $x = 1.3$, which converges, gets 0.006.

How far an answer can be trusted

query_report(m, z₀, ρ, ps, st) answers the query and gathers these checks into one QueryReport: whether the solve converged to a stable root, the other stable roots reached from random starts and where the answer stands among them, the conditioning of the free-coordinate Jacobian (the test implicit_pullback applies, and the smallest curvature), the round trip, the ridge distance (with the dense solver), and, with reference = coverage_reference(...), the coverage p-value. r.flags lists what failed (:not_converged, :unstable, :other_roots, :not_lowest, :ill_conditioned, :round_trip, :off_ridge, :atypical); trusted(r) is true when it is empty. A round trip that finds no stable point, and a ridge distance of Inf, are always flagged; a finite distance only against round_trip_tol or ridge_tol, tolerances in data units that the caller supplies.

Samples, not only modes

The root finders return modes. implicit_sample(m, z₀, ρ, ps, st; n, steps, resample) draws n samples of the free coordinates given the hard-clamped ones (ρ = Inf) with the reverse diffusion, replacing the clamped coordinates at every step; resample > 0 adds RePaint's harmonisation, which a strongly coupled clamp needs. With noise blocks, clamp_level = tmin holds the clamped blocks clean instead, and guidance = w adds classifier-free guidance ((Ho and Salimans, 2022)). On MNIST, holding the label clean is what makes label → image samples depend on the label (85.5% recognised, against 10.5% with replacement).

Two options make the sampler work on a product of relations, whose summed scores are not the scores of the noised product ((Du et al., 2023)). corrector = :langevin or :mala adds Markov chain steps at every noise level ((Roberts and Tweedie, 1996)); they fix the shape of each mode. smc = true also weights and resamples the chains, a sequential Monte Carlo sampler over the noise levels ((Del Moral et al., 2006), (Skreta et al., 2025)); it fixes how the mass is split between modes, which local moves cannot. Both :mala and smc need log-densities (mixtures, kernel estimates, energy networks).

Differentiating through inference

With ChainRulesCore loaded (Zygote loads it), implicit_infer has an rrule whose pullback is implicit_pullback: Zygote.gradient(ps -> loss(first(implicit_infer(m, z₀, ρ, ps, st)).z), ps) works without unrolling the solver, and gives gradients for z₀ and ρ too.

Products of relations

ProductRelation(m₁, m₂; weights) pools relations on the same variables: its field is the weighted sum, its stable roots are where all relations hold, e.g. the intersections of a learned circle and a learned line. Every query function and the adjoint accept it.

Relations on different variables compose with pairs relation => coordinates, and a separator is a relation with weight $-1$ on the shared coordinates: ProductRelation(mA => [1, 2], mB => [2, 3], mS => [2]; weights = [1, 1, -1]) is $p_A(x, y)\,p_B(y, z)/p_S(y)$. Without the separator the shared marginal counts twice: on a chain whose two modes carry 0.8 and 0.2, the Laplace mixture weighs them 0.80 / 0.20 with it and 0.94 / 0.06 without.

Longer chains need a separator on every inner variable. In examples/chain.jl, a bit passed along three to five links (the outer relations share no variable) is answered with every path of the bit at its exact probability. The adjoint through a chain of three to five curves matches finite differences of the solve, and the smoothing bias adds up along the chain.

API

The relation, the solver and the adjoint

VariationalDiffusion.FieldNodes — Type
FieldNodes(t, ε, w)

The fixed quadrature nodes $(t_k, \varepsilon_k, w_k)$ that turn RED-Diff's stochastic gradient into a deterministic vector field. Fixing them is sample-average approximation: the field, its root and its Jacobian become ordinary deterministic objects, which is what the implicit function theorem needs. Build them with field_nodes or noisefree_nodes.

source
VariationalDiffusion.ImplicitDiffusion — Type
ImplicitDiffusion(predictor, nodes; λ = 1.0, batched = false)

The relation $R_\theta = \{z : r(z) = 0\}$ defined by a noise predictor and a fixed node set, with RED-Diff's weighting $\lambda_t = \lambda\sigma_t/\alpha_t$.

With batched = true all nodes of one noise level go through the network in one call, as the columns of an $n\times B$ matrix at a scalar time: one call per level instead of one per node, for the field and for every derivative. The predictor must then accept a matrix input (a Lux Chain of Dense layers does, with an input function that repeats the time embedding across columns). The answers are the same either way (implicit.md §6).

Noise blocks. For a predictor with blocks (NoisePredictor), levels gives each block's noise level at the nodes: nothing for a block annealed with the nodes (at each node's level), a number for a block held at that level at every node. A clamped block held at the schedule's tmin is a clean condition, so the field on the free blocks is the smoothed conditional score, $\nabla\log p_t(z_F \mid z_C)$, instead of the joint score at a common level (conditional_levels; implicit.md §9). The node input is then $\alpha_k \odot z + \sigma_k \odot \varepsilon_k$ with α and σ per coordinate, and the weight of coordinate $i$ is $\lambda\,(\sigma_t^2/\alpha_t^2)\,\alpha_i/\sigma_i$, which is RED-Diff's $\lambda\sigma_t/\alpha_t$ on the annealed blocks and keeps the field of an energy network a gradient.

source
VariationalDiffusion.ImplicitProx — Type
ImplicitProx(nodes; λ = 1.0, tol = 1e-9, maxiters = 200, step = 0.05, message = :point,
             batched = false, linsolve = DenseSolve())

The deterministic implicit solver as a DiffusionFactor's inversion, in place of REDDiff:

f = DiffusionFactor((x = 1, y = 1), pred; prox = ImplicitProx(field_nodes(rng, 2)))

Inversion is then implicit_infer on the factor's state space with the polarity's precisions (Inf = hard clamp). Compared with RED-Diff it is deterministic, it reports convergence and stability (implicit_solution), its free energy is deterministic, and it has a backward pass (implicit_factor_pullback). nodes must have the factor's state dimension.

message chooses what invert and factor_message return on the target channel:

  • :point — a DiracBelief at the answer;
  • :gaussian — a GaussianBelief from the Laplace approximation at the answer (laplace_belief): the posterior marginal from invert, and from factor_message the same with the target's own anchor divided out, so that a variable does not count its incoming belief twice. Use λ = density_lambda(schedule, nodes) for calibrated covariances.

batched and linsolve are passed on to ImplicitDiffusion and to the solver: one network call per noise level, and the matrix-free KrylovSolve for factors with many free coordinates.

source
VariationalDiffusion.ImplicitSolution — Type
ImplicitSolution

What inference returns: the state z, the residual norm on the free coordinates, the number of iterations, and two flags.

  • converged — the residual is below tolerance. If not, z is still returned (an anytime answer) but the implicit function theorem does not apply at it.
  • stable — the symmetric part of $J_{FF}$ is positive definite: a strict local minimum of the energy when the field is a gradient. A root that is not stable is a saddle or a maximum of the energy — on the zero set of $r$, but not a member of the relation one wants.
source
VariationalDiffusion.conditional_levels — Method
conditional_levels(pred::NoisePredictor, ρ; level = tmin) -> Vector

The block levels (ImplicitDiffusion's levels) of a query with precisions ρ: every block whose coordinates are all hard-clamped is held at level (by default the schedule's tmin, a clean condition), the others are annealed (nothing).

source
VariationalDiffusion.default_tolerance — Method
default_tolerance(m, z, ps) -> Real

The solvers' default tolerance for the relation m with parameters ps at states like z: $\max(10^{-9},\ 30\,\epsilon)$, with $\epsilon$ the machine epsilon of the arithmetic the field is computed in. On a device (NoisePredictor's device) that is the parameters' type, since the inputs are converted to it. A Float32 network's field has a rounding floor of 10 to 15 $\epsilon$ relative (measured on MNIST with 10 and with 784 free coordinates, implicit.md §11), so the tolerance, about $3.6\cdot10^{-6}$, sits a factor of two to three above it. On the CPU, Julia promotes Float32 parameters against Float64 states to Float64, so the tolerance stays $10^{-9}$.

source
VariationalDiffusion.field_nodes — Method
field_nodes(rng, n; levels = range(0.002, 0.05; length = 8), samples = 4, antithetic = true,
            orthogonal = false)

samples noise draws at each noise level in levels, equally weighted. With antithetic each draw $\varepsilon$ is paired with $-\varepsilon$: the control-variate term $-\sum w_k\lambda_k\varepsilon_k$ of the field then cancels exactly, so the field has no constant tilt. With orthogonal = true, draws within each block of up to n samples at a given noise level are QR-orthogonalised while preserving their radial $\chi_n$ norms, reducing quadrature variance across spatial directions.

The levels matter more than anything else here. A relation is only recovered at noise scales below its own curvature scale: the default stops at $t = 0.05$ ($\sigma_t\approx0.16$). RED-Diff's training-time range $t\in[t_{\min},1]$ smooths a unit circle into a blob whose density peaks at the centre (Implicit Diffusion Learners.md §5).

source
VariationalDiffusion.implicit_infer — Method
implicit_infer(m, z₀, ρ, ps, st; z_init = z₀, tol = default_tolerance(m, z₀, ps), maxiters = 200, step = 0.05,
               linsolve = DenseSolve(), continuation = (), method = :auto,
               radius = nothing) -> (ImplicitSolution, st)

Find a stable root of the residual on the free coordinates, warm-started from z_init.

Each iteration takes a Newton step $-J_{FF}^{-1}r_F$ with backtracking on $\|r\|$ when the symmetric part of $J_{FF}$ is positive definite, and otherwise a descent step $-\texttt{step}\cdot r_F$ along the field (energy descent when the field is a gradient). Newton is only trusted in locally convex regions so that it is not drawn to saddles and maxima.

z₀ holds the clamp values on hard coordinates and the soft targets elsewhere; ρ is the precision vector, Inf for a hard clamp (cf. precision_vector).

linsolve chooses the linear algebra: DenseSolve forms $J_{FF}$; KrylovSolve takes the Newton step by GMRES on Jacobian-vector products and tests stability by Lanczos, without forming it (for many free coordinates).

continuation = (c₁, c₂, …) solves first with every node's noise level multiplied by c₁ (capped at 1), then by c₂, …, each solve started from the previous answer, and last with m itself: a continuation from a smoother relation to the actual one. A smoother field has fewer, wider basins, so a start far from every basin (a blank image, say) is carried into one before the sharp field is solved. Each stage takes at most maxiters iterations and is used whether or not it converged; iters counts all of them. For an ImplicitDiffusion.

Trust region. method = :trust_region replaces the Newton-or-descent iteration by a trust-region method on a merit function, which converges from far starts and in non-convex regions where Newton-or-descent stalls (implicit.md §11). The default, method = :auto, uses it when the relation has an energy and Newton-or-descent otherwise; :newton forces the latter.

  • with an energy (EnergyNetwork, GaussianMixtureEps, products of them) the merit is the query's energy $Q(z) = U(z) + \tfrac12\lVert\rho_F(z_F - z_{0,F})\rVert^2$, whose gradient is the residual and whose Hessian is $\operatorname{sym}J_{FF}$: every accepted step lowers $Q$, so the iteration is not drawn to saddles or maxima;
  • without one it is $\tfrac12\lVert r_F\rVert^2$ with the Gauss–Newton model (Levenberg–Marquardt), which can stop at a minimum of the residual that is not a root (then converged is false).

Each step minimises the quadratic model within a ball of radius $\Delta$: exactly from an eigendecomposition with DenseSolve, by Steihaug's truncated conjugate gradients on Hessian-vector products with KrylovSolve. $\Delta$ starts at radius (default $0.1\sqrt{|F|}$) and grows or shrinks with the ratio of actual to predicted decrease. step is not used.

tol is relative, $\lVert r_F\rVert \le \mathrm{tol}\,(1 + \lVert z\rVert)$, and defaults to default_tolerance: $10^{-9}$ in Float64, about $3.6\cdot10^{-6}$ for a Float32 network on a device.

source
VariationalDiffusion.implicit_pullback — Method
implicit_pullback(m, sol, z₀, ρ, z̄, ps, st; linsolve = DenseSolve()) -> (z₀ = z̄₀, ρ = ρ̄, ps = ps̄)

Reverse-mode derivative of $z^\star(z_0,\rho,\theta)$ by the adjoint method. Given the cotangent z̄ $= \partial\ell/\partial z^\star$ of a downstream loss, solve

\[J_{FF}^\top\lambda = -\bar z_F\]

once, and read every gradient off $\lambda$:

w.r.t.cotangent
parameters $\theta$$\sum_k w_k\lambda_{t_k}\,(\partial_\theta\varepsilon_\theta(x_k,t_k))^\top\lambda$ (via epsilon_vjp_params)
hard-clamped inputs $z_{0,C}$$\bar z_C + J_{FC}^\top\lambda$
soft targets $z_{0,F}$$-\rho_F^2\odot\lambda$
precisions $\rho_F$$2\rho_F\odot(z^\star_F - z_{0,F})\odot\lambda$

No noise has to be stored from the forward pass: the nodes are part of the model, and the Jacobians are evaluated at $z^\star$ afresh. Throws if sol did not converge, because the implicit function theorem says nothing about a point that is not a root (Backpropagation through Implicit Inference.md §6).

With linsolve = KrylovSolve() the adjoint system is solved by GMRES on vector-Jacobian products, so $J_{FF}$ is never formed; it then warns if GMRES stops short of its tolerance.

source
VariationalDiffusion.implicit_residual — Method
implicit_residual(m, z, z₀, ρ, ps, st) -> (r, st)

$r = g_\theta(z) + \rho^2\odot(z - z_0)$ on the free coordinates (finite ρ), zero on the hard-clamped ones ($\rho_i = \infty$), where the constraint is $z_i = z_{0,i}$ instead.

source
VariationalDiffusion.noisefree_nodes — Method
noisefree_nodes(n, t)

One noise level, no noise: $g(z) = \lambda_t\,\varepsilon_\theta(\alpha_t z, t)$. This is the deterministic relaxation: a root satisfies $\hat x_0(\alpha_t z) = z$ on the free coordinates — a fixed point of the Tweedie denoiser, i.e. a deep equilibrium model whose layer is the denoiser (Deterministic Relaxation.md).

source
VariationalDiffusion.prior_field — Method
prior_field(m, z, ps, st) -> (g, st)

$g_\theta(z) = \sum_k w_k\lambda_{t_k}\bigl(\varepsilon_\theta(\alpha_k z+\sigma_k\varepsilon_k,t_k)-\varepsilon_k\bigr)$.

source
VariationalDiffusion.prior_jacobian — Method
prior_jacobian(m, z, ps, st; coords = :) -> Matrix

$\partial g/\partial z = \sum_k w_k\lambda_{t_k}\alpha_{t_k}\,\partial_x\varepsilon_\theta$. Symmetric iff the field is (locally) a gradient.

With coords (indices or a mask) only the block $J_{II}$ on those coordinates is computed, at a cost that grows with their number, not with the dimension (see epsilon_jacobian_block). The solver, the Laplace approximation and the adjoint only ever need the block on the free coordinates, so a query with ten free coordinates in a 794-dimensional state costs ten derivative passes per noise level, not 794.

source
VariationalDiffusion.prior_vjp — Method
prior_vjp(m, z, w, ps, st) -> Vector

The vector–Jacobian product $(\partial g/\partial z)^\top w$ of the field, without forming the Jacobian: one input VJP per noise level (epsilon_vjp_input). The adjoint uses it for the cotangent of the clamped inputs, $J_{FC}^\top\lambda$.

source

Answers and uncertainty

VariationalDiffusion.calibrate_temperature — Method
calibrate_temperature(answers, classes, truth; bounds = (1e-3, 10), scale_bounds = (1e-4, 10))
    -> (temperature = τ, cov_scale = κ, nll)

The temperature τ and covariance scale κ of categorical_answer that maximise the likelihood of the true classes truth (indices into the columns of classes) on held-out answers: answers are the mixtures (MixtureBeliefs on the label coordinates) of held-out queries. Alternating golden-section searches over $\log\tau$ and $\log\kappa$ within the bounds (τ in the units of the codes). nll is the mean negative log-likelihood at the optimum. Pass scale_bounds = (1, 1) to fit τ alone.

source
VariationalDiffusion.categorical_answer — Method
categorical_answer(mix::MixtureBelief, classes::AbstractMatrix; labels = nothing, radius = nothing,
                   temperature = nothing, cov_scale = 1) -> (belief, unassigned, assignment)

A mixture answer on label coordinates read as a class: each component is assigned to the nearest column of classes (the class codes, e.g. scaled one-hot vectors, one column per class), and the weights of the components assigned to a class are summed.

  • belief: a CategoricalBelief over the classes (with labels), normalised over the assigned mass; nothing if no component is assigned. With temperature = τ it is instead the posterior of the code given each component's position and covariance, $p(k) = \sum_j w_j\,\mathcal N(c_k;\ \mu_j,\ \kappa\Sigma_j + \tau^2 I) / \sum_l \mathcal N(c_l;\ \mu_j,\ \kappa\Sigma_j + \tau^2 I)$ with $\kappa$ = cov_scale, which gives graded probabilities also when the query has a single root (a root between two codes splits its mass between them). Fit τ and κ on held-out answers with calibrate_temperature: κ < 1 sharpens a Laplace covariance that is too wide, τ widens one that is too narrow.
  • unassigned: the mixture weight of the components at least radius from every class code, reported rather than dropped. A root between the codes is an answer the relation gives but no class describes (a smoothing artefact, or a query outside the data).
  • assignment: the class of each component, 0 for unassigned.

radius defaults to half the smallest distance between two class codes, so that the regions do not overlap. The means are compared, not the Gaussians: a component's own spread does not split its mass between classes.

source
VariationalDiffusion.code_posterior — Method
code_posterior(g, classes, τ; cov_scale = 1) -> Vector

The probability of each class code (column of classes) given a Gaussian answer g on the label coordinates, with the codes equally likely a priori: $\propto \mathcal N(c_k;\ \mu,\ \kappa\Sigma + \tau^2 I)$, $\kappa$ = cov_scale.

source
VariationalDiffusion.density_lambda — Method
density_lambda(schedule, nodes::FieldNodes) -> λ

The weighting $\lambda = 1 / \sum_k w_k\,\sigma_k^2/\alpha_k^2$ under which the implicit learner's prior term is a weighted average of smoothed negative log-densities, $U(z) \approx \sum_k \bar w_k\,(-\log p_{t_k}(\alpha_k z + \sigma_k\varepsilon_k))$ with $\sum_k \bar w_k = 1$: a proper negative log-density, in the same units as the clamp's Gaussian log-likelihood. With it, implicit_laplace returns covariances in data units. For queries whose coordinates are all hard inputs or free outputs ($\rho_i \in \{0, \infty\}$), λ does not change the answers, only the curvature; with soft evidence it sets the balance between prior and evidence.

source
VariationalDiffusion.implicit_classify — Method
implicit_classify(m, z₀, ρ, ps, st; label, classes, labels = nothing, radius = nothing,
                  starts = class codes, kwargs...) -> (answer, st)

Classification as a query: the coordinates label (indices) are free and read as a class, the others are clamped or soft as ρ says. Runs implicit_mixture on the label coordinates, by default started from every class code (the columns of classes placed in the label coordinates, the rest of z₀ unchanged), and reads the answer with categorical_answer (with radius, temperature and cov_scale). kwargs go to the solver (maxiters, linsolve, …).

answer holds belief (a CategoricalBelief, or nothing), unassigned, assignment and mixture, the Gaussian mixture it was read from. The class probabilities are the Laplace masses of the roots, so they are calibrated only when the relation has an energy (an EnergyNetwork or a closed-form mixture) and λ is density_lambda; without an energy every root weighs the same.

source
VariationalDiffusion.implicit_laplace — Method
implicit_laplace(m, sol, ρ, ps, st) -> (mean, cov, free)

The Laplace approximation of the query at the answer sol: mean sol.z, and on the free coordinates the covariance

\[\Sigma_{FF} = \bigl(\operatorname{sym} J_{FF}\bigr)^{-1}, \qquad J_{FF} = \partial_{z_F} r_F(z^\star) = \partial_{z_F} g_F + \operatorname{diag}(\rho_F^2),\]

the inverse curvature of the query's energy (for an energy-parametrised predictor $J_{FF}$ is already symmetric; otherwise its symmetric part is used). Hard-clamped coordinates have zero variance. free marks the free coordinates.

Non-conservative fields. For a noise predictor that is not an energy network, the field need not be a gradient, $J_{FF}$ need not be symmetric, and there need be no energy whose curvature this is: $(\operatorname{sym} J_{FF})^{-1}$ is then a local-curvature (stability) measure, not a Laplace posterior covariance. A warning is given when the antisymmetric part exceeds $10^{-6}$ of $\lVert J_{FF} \rVert$ (review of 2026-11-04).

Units. The energy is the clamp $\tfrac12\lVert P(z - z_0)\rVert^2$ (a Gaussian log-likelihood with precision $\rho^2$) plus the prior term, a weighted sum of smoothed negative log-densities scaled by m.λ. cov is a calibrated posterior covariance when that prior term is a calibrated negative log-prior. Build the model with λ = density_lambda(schedule, nodes) for that: the covariance is then calibrated to the density smoothed at the field's noise levels (wider than the data distribution by those levels). With another λ it is a relative measure of how sharply the answer is determined. Throws unless sol converged to a stable answer.

source
VariationalDiffusion.implicit_mixture — Method
implicit_mixture(m, z₀, ρ, ps, st; coords = free, linsolve = DenseSolve(), kwargs...) -> (MixtureBelief, st)

Every answer to a query as one belief: a mixture of the Laplace Gaussians at the stable roots that implicit_roots finds (kwargs are passed on), marginalised onto coords.

The weights are the Laplace estimates of each answer's probability mass, $w_k \propto e^{-E(z_k)}\,\det(\Sigma_k)^{1/2}$, with $E$ the query's energy (the relation's plus the clamp's) and $\Sigma_k$ the full free-coordinate covariance, when the relation has an energy: an EnergyNetwork, a closed-form GaussianMixtureEps, or a product of them. Otherwise the field determines no relative heights and the weights are equal. Build the model with density_lambda so that the energy is a calibrated negative log-density. Throws if no stable answer is found.

With linsolve = KrylovSolve() the roots are found matrix-free, the covariances are computed on coords only, and $\log\det\Sigma_k$ is a stochastic Lanczos estimate (exact when at most probes coordinates are free).

source
VariationalDiffusion.implicit_roots — Method
implicit_roots(m, z₀, ρ, ps, st; nstarts = 16, spread = 1.0, starts = nothing,
               rng = Xoshiro(0), unique_tol = 1e-3, kwargs...) -> (Vector{ImplicitSolution}, st)

All distinct stable answers to a query that a set of starting points reaches. Starts are z₀ itself and nstarts perturbations of it on the free coordinates (normal, standard deviation spread), or the vectors in starts; hard-clamped coordinates stay fixed. Each start is solved with implicit_infer (kwargs are passed on); converged, stable solutions are kept, and two are the same answer if they differ by less than unique_tol (relative).

The answers are ordered by the query's energy $U(z) + \tfrac12\lVert P(z - z_0)\rVert^2$ when the predictor has an energy, i.e. is an EnergyNetwork or a closed-form GaussianMixtureEps (lowest first, i.e. the most plausible answer first), and otherwise in the order found. There is no guarantee that every stable answer is found; more starts and a larger spread find more.

source
VariationalDiffusion.laplace_belief — Method
laplace_belief(m, sol, ρ, ps, st; coords = free, linsolve = DenseSolve()) -> GaussianBelief

The Laplace approximation at the answer sol (implicit_laplace) as a GaussianBelief, marginalised onto coords (indices or a mask; default: all free coordinates). Hard-clamped coordinates have no variance and cannot be among coords. Calibrated in data units when the model is built with density_lambda.

With linsolve = KrylovSolve() only the marginal is computed, one conjugate-gradient solve per coordinate in coords, so a label's covariance can be read off a query whose image is free.

source
VariationalDiffusion.ridge_residual — Method
ridge_residual(m, z, ps, st; codim) -> (NamedTuple, st)
ridge_residual(m, sol::ImplicitSolution, ρ, ps, st; codim = count(!isinf, ρ)) -> (NamedTuple, st)

Is the point z on the relation, judged by the field at z alone? The relation is the ridge of the smoothed density, of codimension codim: across it ($k$ = codim directions) the field pulls back, along it the field may still drift, since the density varies along the relation. Let $S = \operatorname{sym}\,\partial g/\partial z$ be the symmetric part of the field's full Jacobian (prior_jacobian, all coordinates, clamped ones included), $V$ its eigenvectors of the $k$ largest eigenvalues $\Lambda$ (the normal directions) and $g$ the field (prior_field). Then

\[\delta = V\Lambda^{-1}V^\top g(z)\]

is a Newton step projected on the normal directions (subspace-constrained mean shift, Ozertem & Erdogmus 2011, projects the mean-shift vector the same way), and its length is the first-order distance to the ridge, in data units: exact for a Gaussian (an affine field with a symmetric Jacobian), and unchanged by the field's weight λ. The tangential part of $g$ is dropped, so an answer on the relation where the density changes along it is not rejected; nor is one where the relation is parallel to a clamped coordinate, where round_trip fails (calibration.md, "Remedies").

For an answer sol to a query with precisions ρ, the codimension is by default the number of free coordinates: the relation is taken to determine them from the clamped ones.

Returns (distance, residual, curvature, gap, normal):

  • distance $\lVert\delta\rVert$; Inf when a normal curvature is not positive ($\Lambda_{\min} \le 0$): the field does not pull back across in every normal direction, so z is on no ridge of this codimension;
  • residual $\lVert V^\top g\rVert$, the normal component of the field, in the field's units;
  • curvature $\Lambda$, ascending;
  • gap the smallest normal minus the largest tangential eigenvalue (Inf when codim is the dimension). Where it is small the normal directions are not well determined (a crossing of two branches, an isotropic blob), and so neither is distance;
  • normal $V$, the normal directions as columns.

codim = 0 gives distance = 0. Soft evidence that pulls an answer off the relation shows in distance as it does in the round trip's. The Jacobian is formed in full, one derivative pass per coordinate and noise level, so this is for states of moderate dimension; there is no matrix-free version. The answer is called on the relation when distance is below a tolerance in data units, above the smoothing bias ($O(\sigma_t^2\kappa)$, implicit.md).

source
VariationalDiffusion.round_trip — Method
round_trip(m, sol::ImplicitSolution, z₀, ρ, ps, st; kwargs...) -> (NamedTuple, st)

Is the answer sol to the query (z₀, ρ) on the relation, or a projection onto it? The query is reversed: the coordinates it clamped hard ($\rho_i = \infty$) are freed, every other one is clamped hard at the answer, and implicit_infer solves back from the answer itself (kwargs are passed on). An answer on the relation is already a root of the reversed query, so the clamped coordinates do not move; a query off the relation ($x = 1.3$ on the unit circle) has a root on the smoothed density's ridge, from which the reversed query walks back onto the relation ($x \approx 1$).

Returns (distance, solution): distance is $\lVert z'_C - z_{0,C}\rVert$, the displacement of the clamped coordinates $C$, in data units, and solution the reversed query's ImplicitSolution. When it did not converge or is not stable, distance is Inf: no stable point of the relation was found through the answer. A query without hard clamps has nothing to send back and returns distance = 0 and solution = nothing.

Soft evidence ($0 < \rho_i < \infty$) pulls an answer off the relation towards its targets; the reversed query clamps those coordinates at the answer, so distance then measures how far the evidence pulled, a conflict between the evidence and the relation, as well as a query off it.

The answer is called on the relation when distance is below a tolerance in the data's units, such as a few noise widths of the data. The smoothing bias of the field moves even an exact answer a little ($O(\sigma_t^2\kappa)$, implicit.md), so the tolerance must be above it.

source

Matrix-free linear algebra

VariationalDiffusion.DenseSolve — Type
DenseSolve()

Form the free-coordinate block $J_{FF}$ (prior_jacobian with coords = free) and use dense factorisations: exact, and the right choice while few coordinates are free (one derivative pass per free coordinate and noise level). The default.

source
VariationalDiffusion.KrylovSolve — Type
KrylovSolve(; rtol = 1e-8, maxiters = 200, restart = 40, lanczos = 30, probes = 8, seed = 0)

Matrix-free linear algebra, for queries with many free coordinates (generating an image):

  • the Newton step and the adjoint solve $J_{FF}d = -r_F$ and $J_{FF}^\top\lambda = -\bar z_F$ by restarted GMRES (relative tolerance rtol, at most maxiters products, restarts every restart), on Jacobian-vector products (prior_jvp) and vector-Jacobian products (prior_vjp);
  • stability ($\operatorname{sym}J_{FF}\succ0$) is decided by the smallest Ritz value of lanczos Lanczos steps on the symmetric part;
  • the Laplace approximation solves for the covariance on the requested coordinates only, by conjugate gradients, and estimates $\log\det$ by stochastic Lanczos quadrature with probes Rademacher probes of lanczos steps each.

Lanczos and the probes draw from Xoshiro(seed), so the answers are deterministic. With at most lanczos free coordinates the stability test is exact, and with at most probes the log determinant is too (the probes are then the unit vectors, each with full Lanczos depth regardless of the lanczos setting). Otherwise both are estimates: the smallest Ritz value approaches $\lambda_{\min}$ from above, so a barely unstable point can be reported stable (krylov.md §3).

source

Many queries at once

VariationalDiffusion.implicit_classify — Method
implicit_classify(m::ImplicitDiffusion, Z₀::AbstractMatrix, ρ, ps, st; label, classes,
                  labels = nothing, radius = nothing, unique_tol = 1e-3, kwargs...)
    -> (Vector{answer}, st)

implicit_classify for many queries at once, the columns of Z₀ (all with the same ρ). Every query is started from every class code, and all starts of all queries are solved together by implicit_infer_many (kwargs go to it); the Laplace covariances and the energies of the roots are computed in batches as well. Each answer is what the single-query method returns: the same roots, weights and class probabilities.

source
VariationalDiffusion.implicit_infer_many — Method
implicit_infer_many(m::ImplicitDiffusion, Z₀::AbstractMatrix, ρ, ps, st; Z_init = Z₀,
                    tol = default_tolerance(m, Z₀, ps), maxiters = 200, step = 0.05, method = :auto,
                    radius = nothing) -> (Vector{ImplicitSolution}, st)

implicit_infer for many queries at once: column b of Z₀ is a query (its clamps and soft targets), all with the same precisions ρ, warm-started from column b of Z_init. Returns one solution per column, the same ones implicit_infer returns query by query (to the solver's tolerance).

Every iteration evaluates the field of all unfinished queries in one network call per noise level (with m.batched; otherwise one call per node input), and their free-coordinate Jacobian blocks with one input JVP per free coordinate and noise level (a VJP when the predictor has an AD backend). Each query takes its own Newton or descent step, backtracks on its own residual, and leaves the batch when it converges. The Jacobian blocks are dense, so this is for queries with few free coordinates, such as classification (the image clamped, ten label coordinates free); for many free coordinates solve the queries one by one with KrylovSolve.

method is that of implicit_infer: by default (:auto) the trust-region method for a relation with an energy, each query with its own radius and acceptance test, and Newton-or-descent otherwise. The trial points of all queries are evaluated together as well (residual and energy, one call per noise level each).

source

What a query costs

VariationalDiffusion.break_even_classes — Method
break_even_classes(m::ImplicitDiffusion, nfree, units; draws, kwargs...) -> K

The number of classes above which root finding with nfree free coordinates is predicted to be cheaper than the diffusion classifier with draws noise draws per class: root_cost(m, nfree, units; kwargs...) / (draws · units.epsilon). Root finding's cost grows with the free coordinates, the classifier's with the classes. With one-hot labels (nfree = K) the crossing may never come; with a label code of fixed dimension it always does, at some K.

source
VariationalDiffusion.classifier_cost — Method
classifier_cost(nclasses, draws, units) -> seconds per query

The predicted time per query of the diffusion classifier (Li et al. 2023) on the same network: one ε evaluation per class and noise draw, $K M c_\varepsilon$, from the units of pass_costs.

source
VariationalDiffusion.pass_costs — Method
pass_costs(pred::NoisePredictor, X, t, ps, st; directions = 1, reps = 3)
    -> (epsilon, energy, derivative)

The measured cost, in seconds per column, of the three passes a query makes through the network, at the batch width of X (one column per input, as the batched solvers call it):

  • epsilon: $\varepsilon_\theta$ (epsilon);
  • energy: the energy of an EnergyNetwork (nothing for other predictors);
  • derivative: one input derivative per column, measured on directions copies of X side by side as implicit_infer_many stacks the free coordinates: a VJP with an AD backend (rows of the Jacobian), a JVP without.

Each pass runs once to compile, then reps times; the fastest is kept. With a device the transfer of the result back is included, which is also where the device synchronises.

source
VariationalDiffusion.root_cost — Method
root_cost(m::ImplicitDiffusion, nfree, units; iterations, starts = 1, method = :auto, backtracks = 1)
    -> seconds per query

The predicted time per query of root finding with implicit_infer_many: starts solves of iterations iterations each, with nfree free coordinates and the relation's field nodes, from the per-column units of pass_costs. Per iteration and node, one ε evaluation and nfree input derivatives (the Jacobian block, once more at the end for the stability test); the trust region (method, as in implicit_infer) adds the energy at each trial point, Newton-or-descent backtracks trial residuals per iteration. The iterations are an input: measure them on a few queries, they vary little across queries of one kind.

source

Coverage

VariationalDiffusion.coverage — Method
coverage(ref::CoverageReference, z) -> NamedTuple
coverage(ref::CoverageReference, sol::ImplicitSolution) -> NamedTuple
coverage(ref::CoverageReference, Z::AbstractMatrix) -> Vector{NamedTuple}

How typical the answer z is of the data, by three scores and their p-values:

fieldscore
distance, p_distancedistance to the nearest data point
reconstruction, p_reconstruction$\tfrac1K\sum_k \lVert\hat x_0(\alpha_t z + \sigma_t\xi_k) - z\rVert^2$, Tweedie's denoiser at level t
energy, p_energy$\tfrac1K\sum_k E(\alpha_t z + \sigma_t\xi_k, t)$, $E = -\log p_t$; nothing without an energy
pthe combined p-value: the smallest one times the number of scores (Bonferroni), at most 1

Each p-value is $(1 + \#\{\text{reference scores} \ge s\}) / (1 + N_c)$: the fraction of calibration points at least as atypical. For an in-distribution point exchangeable with the calibration points it is uniform, so p < 0.05 flags about 5% of them by chance and an answer outside the data with high probability. A matrix scores each column.

The scores read the model at the answer and the data near it; they do not certify the answer. A root on the relation can still be the wrong branch (T1), and an answer of the smoothed relation sits slightly off the data (T2), which the reconstruction score sees.

For a classification answer pass unassigned, the mass of categorical_answer that lies between the class codes: it is returned, and flagged is true when p < alpha or at least half the answer is unassigned, so that a root between two codes is flagged rather than read as a class.

source
VariationalDiffusion.coverage_reference — Method
coverage_reference(m::ImplicitDiffusion, data, ps, st; t = 0.05, draws = 8,
                   calibration = nothing, ncal = 500, scale = 1, rng = Xoshiro(0))
    -> CoverageReference

Scores of in-distribution points, against which coverage ranks an answer. data ($\dim z\times N$, one point per column) is the set the nearest-neighbour distance is measured to, normally the training data.

  • calibration: held-out points ($\dim z\times N_c$) whose scores are the reference. By default ncal columns of data are used, each with its own column left out of the nearest-neighbour search. Held-out points are better: a network scores its own training points too well, which makes every answer look slightly atypical.
  • t, draws: the noise level of the reconstruction and energy scores, and the number of noise draws they average over. The draws are fixed, the same for the reference and every query, so the scores are deterministic.
  • scale: a scalar or one value per coordinate; distances are measured in (z - x) ./ scale. Use it when coordinates have different units (pixels and a label, say).

Cost: draws predictor calls per calibration point (one call with batched = true), and a pass over data for each distance.

source

The query report

VariationalDiffusion.QueryReport — Type
QueryReport

What query_report found about one answer to a query. Fields:

fieldmeaning
solutionthe answer, an ImplicitSolution
converged, stablefrom the solve
rootsthe distinct stable roots reached from other starts (implicit_roots), lowest energy first when the relation has one
rankthe answer's position in roots, 0 if it is not among them
conditioning(cond, lambda_min): $\operatorname{cond}_1 J_{FF}$ (nothing with KrylovSolve) and the smallest eigenvalue of $\operatorname{sym} J_{FF}$
round_tripthe result of round_trip, nothing for an answer that did not converge to a stable root
ridgethe result of ridge_residual (codimension the number of free coordinates), nothing for an answer that did not converge to a stable root or with KrylovSolve
coveragethe result of coverage, nothing without a reference
flagswhat failed, a vector of symbols (below)

Flags: :not_converged, :unstable, :other_roots (more than one stable root was found), :not_lowest (a root found has a query energy below the answer's by more than $\sqrt\epsilon$ relative; only for a relation with an energy), :ill_conditioned ($\operatorname{cond}_1 J_{FF}\,\epsilon \ge 10^{-2}$ with $\epsilon$ of $J_{FF}$'s element type: implicit_pullback refuses such an answer), :round_trip (the reversed query found no stable point, or moved the clamped coordinates by more than round_trip_tol), :off_ridge (the ridge distance is Inf, or above ridge_tol), :atypical (coverage flagged it). isempty(r.flags) (or trusted(r)) when none applies.

source
VariationalDiffusion.query_report — Method
query_report(m, z₀, ρ, ps, st; solution = nothing, z_init = z₀, nstarts = 16, spread = 1.0,
             starts = nothing, rng = Xoshiro(0), unique_tol = 1e-3, reference = nothing,
             alpha = 0.05, round_trip_tol = nothing, ridge_tol = nothing, linsolve = DenseSolve(),
             kwargs...)
    -> (QueryReport, st)

Answer the query (z₀, ρ) and say how far the answer can be trusted. The answer is solution when given, otherwise implicit_infer from z_init. Then:

  1. Other starts. implicit_roots with nstarts, spread, starts, rng and unique_tol: every distinct stable root found, and the answer's place among them. A multivalued relation has several; with an energy, they are ordered lowest first.
  2. Conditioning at the answer: $\operatorname{cond}_1 J_{FF}$, the test implicit_pullback applies (dense only), and the smallest eigenvalue of $\operatorname{sym}J_{FF}$, the curvature of the flattest direction (by Lanczos with KrylovSolve). A small eigenvalue means a wide Laplace covariance along it.
  3. On the relation. round_trip for a converged, stable answer. It is flagged when the reversed query finds no stable point (distance Inf), and otherwise only against round_trip_tol, since the distance is in data units (none by default: the value is reported, not judged). It also fails where the relation is parallel to the clamped coordinates, so a flag there is not proof of an answer off the relation (calibration.md, "Remedies"). With DenseSolve also ridge_residual, the first-order distance to the ridge from the field at the answer alone, with codimension the number of free coordinates; flagged when it is Inf (no ridge of that codimension) and otherwise only against ridge_tol, in data units like round_trip_tol. It forms the full Jacobian, one pass per coordinate and level.
  4. Coverage, when reference (a CoverageReference) is given: coverage of the answer at level alpha.

linsolve and kwargs (tol, maxiters, method, …) go to every solve. The cost is about nstarts + 3 solves and, with DenseSolve, one full Jacobian.

source

Sampling

VariationalDiffusion.implicit_sample — Method
implicit_sample(m, z₀, ρ, ps, st; n = 1, steps = 100, resample = 0, corrector = :none,
                corrector_steps = 1, corrector_step = 0.1, T = 1.0,
                rng = Random.default_rng()) -> (Z, st)

n samples of the free coordinates given the hard-clamped ones, as the columns of Z ($\dim z\times n$): the conditional $p(z_F \mid z_C = z_{0,C})$ of the model's data distribution, where the root finders return its modes. ρ marks the polarity as everywhere else: Inf is clamped to z₀, 0 is free. Soft evidence (finite, positive ρ) is not supported: it needs likelihood guidance, which is approximate for a diffusion model. m is an ImplicitDiffusion or a ProductRelation whose relations share one noise schedule.

The sampler is the reverse VP-SDE discretised by DDPM's ancestral step on a grid from T to the schedule's tmin, uniform in log-SNR (steps steps), using the predictors of m only (not their nodes). After every step the clamped coordinates are replaced by $\alpha_s z_{0,C} + \sigma_s\xi$, the clamp's forward marginal at the new time, and the last step returns Tweedie's denoised estimate with the clamps set exactly. With resample = r each step is repeated r more times after re-noising back to the previous time (RePaint's harmonisation), which corrects the free coordinates' lag behind the clamp at r + 1 times the cost. On the circle a few resamplings are needed: without them the conditional samples are biased towards the centre (sampler.md §3).

Corrector steps. With corrector = :langevin or :mala, every step is followed by corrector_steps Markov chain steps on the free coordinates that target the model's density at the new noise level (for a product, $\prod_i p_{i,u}^{\beta_i}$): a Langevin step $x \leftarrow x + \eta\,s(x) + \sqrt{2\eta}\,\xi$ with $\eta =$ corrector_step $\cdot\,\sigma_u^2$, and with :mala a Metropolis–Hastings test that makes the step exact for that density. :mala needs log-densities: closed-form mixtures and energy networks, or products of them. For a single model the corrector refines a sampler that is already consistent; for a product it is what makes the samples right, because the summed scores of the factors are not the scores of the noised product (Du et al. 2023; product.md §5).

Reweighting. Corrector steps fix the shape of each mode but cannot move mass between modes that a low noise level separates. With smc = true the chains are also weighted, as a sequential Monte Carlo sampler for the sequence of densities $\pi_u \propto \prod_i p_{i,u}^{\beta_i}$ (for a single model, $p_u$): each ancestral step is a proposal, and the chain's weight is multiplied by $\pi_u(x_u)\,q(x_t \mid x_u) / (\pi_t(x_t)\,r(x_u \mid x_t))$, with $q$ the forward noising and $r$ the ancestral step; the chains are resampled (systematically) when the effective sample size falls below n/2, and once more at the end, so Z holds unweighted draws. This is consistent as n grows, whatever the mismatch between summed scores and the noised product (Skreta et al. 2025's Feynman–Kac correctors are this construction in continuous time). It needs log-densities, like :mala. With clamps, the clamped coordinates follow $\alpha_u z_{0,C}$ without noise, so that the targets are the conditionals $\pi_u(x_F \mid x_C = \alpha_u z_{0,C})$. The moves between reweightings must leave each level's density invariant, so smc takes corrector = :none or :mala, and no resample.

Held clamps and guidance. For a predictor with noise blocks, clamp_level = tmin (or another level) holds every fully clamped block at that level instead of replacing it: the network sees the clamp clean, at every step, as in training with independent block times (UniDiffuser), and the reverse diffusion of the free coordinates follows the conditional score $\nabla\log p_t(x_F \mid z_C)$. Blocks that are only partly clamped are still replaced. With guidance = w > 0 the noise prediction is $(1 + w)\,\hat\varepsilon(x, z_C) - w\,\hat\varepsilon(x, \varnothing)$, the unconditional prediction taken with the held blocks at level 1, where they are noise (classifier-free guidance, Ho & Salimans 2022): it sharpens the conditional towards the clamp. Not combined with correctors or smc.

With m.batched all n chains go through the network in one call per step; otherwise one call per chain.

source

Products

VariationalDiffusion.ProductRelation — Type
ProductRelation(relations...; weights = ones, dim = nothing)

Several implicit relations pooled: the field is the weighted sum $\sum_i \beta_i\,g_i$ of the relations' fields, i.e. the field of the product of their densities $\prod_i p_i^{\beta_i}$. Its stable roots are the configurations that satisfy all relations together, e.g. the intersection points of a learned circle and a learned line. The weights $\beta_i$ set how much each relation is trusted (a tempering exponent).

Variables. A plain relation sees the whole joint space. A pair relation => coords sees only the coordinates coords of the joint state, in that order, so relations on different but overlapping variables compose:

# p(x, y, z) = p_A(x, y) p_B(y, z) / p_S(y)
ProductRelation(mA => [1, 2], mB => [2, 3], mS => [2]; weights = [1, 1, -1])

A separator is a relation with a negative weight, normally $-1$ on the shared coordinates: without it the shared marginal is counted once per factor that sees it ($p_S^2$ instead of $p_S$ above), which sharpens it and reweights its modes. Every coordinate must be seen by a relation, and the weights of the relations that see a coordinate must sum to a positive number. dim is the dimension of the joint state; by default the largest coordinate used (or the relations' own dimension when none has coords).

Parameters and states are tuples, one entry per relation, in order: ps = (ps₁, ps₂). Every query function accepts a product: implicit_infer, implicit_roots, implicit_laplace, implicit_mixture, implicit_pullback (which returns a tuple of parameter cotangents), and implicit_sample when all relations share a noise schedule. Products may be nested.

Pooling smoothed densities is exact only at zero noise: at the field's noise levels the product of the smoothed densities is not the smoothed product, so intersections are found with the same kind of smoothing bias as single relations, and sampling needs correction steps (product.md §3, §5).

source