implementation

ExtendedGaussian(μ, Σ, D) (extended_gaussian.jl): a Gaussian made flat along a subspace, the belief type for linear relations. Three kinds of direction in one object: flat (D, a likelihood that says nothing there), hard (null(Σ) outside D, a linear constraint imposed exactly) and Gaussian. Pushforward through affine maps and the product are exact; GaussianBelief, DiracBelief and TrivialBelief are special cases and convert to it.

Sources: Stein & Samuelson, A Category for unifying Gaussian Probability and Nondeterminism, CALCO 2023 (arXiv:2204.14024: subspaces admitted as generalised uniform distributions); Stein & Samuelson, A Categorical Treatment of Open Linear Systems, LMCS 21(3), 2025 (arXiv:2403.03934: extended Gaussian distributions as Willems’ Gaussian linear systems). code: extended_gaussian.jl; Mycelium’s combine methods in lib/Mycelium.jl/src/gaussian.jl; first consumer _improper_split in lib/Lenticulum.jl/src/gaussian.jl. tests: test/extended_gaussian.jl, Mycelium test/runtests.jl (“combine with extended Gaussians”).

Bibliography: Stein & Samuelson 2023 · Stein & Samuelson 2025

Theory (CT-ML wiki): Gaussian Relations

Theory

An extended Gaussian on is the law of with and “uniform” on the subspace : a Gaussian on the quotient . It is the limit of the proper Gaussians with covariance as ( the orthogonal projector onto ). Allowing to be singular adds the other limit: along there is no variance, and the belief imposes exactly ( a basis of those hard directions).

Every belief the Gaussian code handles is one of these:

belief
proper GaussianBelief, full rank
improper GaussianBelief (a likelihood flat along )
DiracBelief(v) (every direction hard)
TrivialBelief()
a hard linear constraint , nothing else on

The canonical form covers the first two and the zero precision; it cannot hold a hard direction (infinite precision). The moment form covers hard directions and not flat ones. The extended form holds both, which is what a factor graph of linear relations needs: a relation that constrains of coordinates sends a message flat in directions, and a noise-free relation imposes its directions exactly.

Pushforward. For with independent, the law of is . The image of a uniform law on is uniform on , so this is exact for every matrix , rectangular or rank-deficient. Marginalising onto coordinates is the pushforward through the selection matrix.

Preimage. For a belief about , the likelihood is again an extended Gaussian on : hard where would have to lie in a hard direction of (), Gaussian with precision and information , flat along and the preimage of ‘s flat directions. With the pushforward it gives the message of every linear relation (HardConstraintFactor, lib/Lenticulum.jl/src/hard_constraint.md).

Product. Two extended Gaussians as potentials on : each is supported on an affine subspace (its hard constraints) and, there, is , constant along . The product is supported on the intersection: stack the constraints into , take a particular solution (refusing the product when none exists: contradictory constraints) and an orthonormal basis of , and write . The precisions add, , , and in the quadratic part is , . Along the product is flat; on the rest it is Gaussian with mean and covariance . Mapped back: , , . No component of can lie along : a vector there has (both are positive semidefinite), so . The product of two extended Gaussians is therefore always one, or a contradiction.

Implementation

  • Normalised representative. The constructor orthonormalises (an SVD, dependent columns dropped) and projects and onto , since nothing along is observable. Two representatives of one law then agree up to the basis of ; the tests compare .
  • Tolerances. An eigenvalue or singular value below counts as zero, the rule _split_moments used before. So a direction with variance below about of the largest is hard: a deliberate choice, since a covariance that small is a constraint in all but name, and treating it as Gaussian is what made the rank-deficient messages of LP-024 wrong. Contradictory constraints are those whose least-squares residual exceeds times the scale of the means.
  • Conversion. extended(::GaussianBelief) splits by the eigendecomposition of (and refuses with a negative eigenvalue beyond rounding, or along : along a flat direction is no limit of Gaussians). GaussianBelief(::ExtendedGaussian) goes back when there is no hard direction, declaring the result improper when .
  • One routine for products and preimages. _from_potential(C, c, η, Λ) builds the extended Gaussian of ; the product and preimage both reduce to it. _independent_join stacks independent beliefs (means, block-diagonal covariances and flat directions).
  • Damping (Mycelium’s damp) is the geometric mixture , the rule canonical damping of a GaussianBelief follows: _conjoin with weights, precisions scaled. Hard constraints of the two sides on one direction are not a contradiction there: with strict = false their values are the weighted least-squares fit, the limit of precisions and , so a hard value moves by the damping weight like any other.
  • Moments. belief_mean and belief_cov return and when the belief is flat nowhere (hard directions have variance zero); along a flat direction there is no mean.
  • Where the product lives. The algebra is here, beside the type (_conjoin); Mycelium’s combine methods call it, for two extended Gaussians or one and a GaussianBelief. A DiracBelief still dominates, as it does against a Gaussian, but is first checked against the other side’s hard constraints.
  • First consumer. _improper_split in Lenticulum’s gaussian.jl, which splits a partly flat input for the constraint, switching, nonlinear and Gaussian factors, is now extended(b). Its outputs are the same up to the basis of the flat directions, which the callers use only through its span. The split itself is _flat_split(η, Λ), which returns the parts without building the type: Lenticulum’s split of a flagged input with a dual and a precision of plain values calls it directly, and the mean carries the duals (P21). Its range check reads ‘s values (stop_gradient): norm of duals with value exactly zero is NaN.

Entropy (2026-11-11)

A belief with hard directions has no density with respect to Lebesgue measure on , but it has one on its support, the affine subspace . support_entropy(b) is the entropy relative to Lebesgue measure there, in orthonormal coordinates:

over the nonzero eigenvalues of . It is the finite part of the entropy of , whose divergent part is . A point gets , the convention Bethe Free Energy already uses for clamped variables. With flat directions the belief is improper and has no entropy; support_entropy throws.

This is a convention, and the Bethe sum is only meaningful if every term follows it: each belief’s entropy relative to its own support. A factor with a smooth density whose input has hard directions works in the input’s coordinates along the other directions. _hard_reduction(b) returns , with the point fixed along the hard directions, an orthonormal basis of the rest, and the Gaussian part in in canonical form, , zero along the flat directions. The hard relation’s own term is in HardConstraintFactor’s note. On trees the result is (Lenticulum test/hard_free_energy.jl), which is what makes the convention the right one there.

Measured

test/extended_gaussian.jl:

  • The product against the limit of proper Gaussians, 135 random pairs (dimension 2–5, 0 to flat and up to hard directions between the two, in random bases). The reference is the product of the Gaussians with covariance , , , in 512-bit arithmetic. Off the product’s flat directions the means and covariances agree to ; along them the reference variance exceeds .
  • Laws: commutativity, associativity, the flat belief as unit, pushforward composing (pushforward(pushforward(b, M₁), M₂) = pushforward(b, M₂M₁)), marginalise as a pushforward.
  • support_entropy against the limit, 60 random beliefs (dimension 1–5, 0 to hard directions): the entropy of minus at in 512-bit arithmetic, with formed there from the generator’s factors (exactly of rank ), agrees to . _hard_reduction, 40 random beliefs with flat and hard directions: orthonormal and orthogonal to the hard directions, the fixed point along them, the precision , zero along the flat directions.
  • Conversions and refusals: round trips with GaussianBelief, the clamp, contradictory constraints, outside , a covariance that is not positive semidefinite.

Derivatives (P4, 2026-11-16)

The type holds any Real (ExtendedGaussian{T<:Real}), so a ForwardDiff dual or a Zygote-tracked float passes through every operation. Before, it held only floats, and the coupled replay of a graph with a HardConstraintFactor failed under both (ForwardDiff at the type bound, Zygote at _independent_join, which wrote into zero arrays and took no tuple).

The rule. Every subspace (a basis of the flat directions, of their complement, of the hard directions, of the range of a covariance or precision, of the solution set of stacked constraints) comes from a decomposition of the values, and its rank is decided there by _eig_tol. The derivative is the one at constant rank. With the value of a matrix and (zero in value, the tangent in derivative), the bases move only into each other:

  • a positive semidefinite with range (eigenvalues ) and null space : , , and , whose derivative is the constant-rank one, (_psd_split, _split_pinv);
  • the column space of : (_orth); the complement of the column space of : (_complement);
  • the least-squares solution and null space of stacked constraints , with at the value: , (_from_potential);
  • the support entropy uses , whose first-order change is that of the nonzero eigenvalues.

No eigenvector or singular vector is differentiated, so repeated eigenvalues are no difficulty; these are the formulas of P25 (LenticulumForwardDiffExt) and P26, written once for both ADs: the decompositions are of values (_value_eigen, _value_svd, marked non-differentiable for ChainRules), and carries the derivative through stop_gradient, which strips a dual and has a zero pullback. With floats is exactly zero and every result is the decomposition’s.

Refusals. A tangent with , and the left and right null spaces at the value, changes the rank at first order: the value jumps there and has no derivative. _check_constant_rank (LenticulumCoreForwardDiffExt) refuses it under ForwardDiff, with the tolerance per partial. A Zygote cotangent cannot see the tangent, so under Zygote the constant-rank derivative is returned without a check.

Where it is used. Mycelium’s combine, damp and belief_distance of a GaussianBelief of duals and an extended Gaussian with hard directions now take the dual side’s extended form (before: an ArgumentError, there being no canonical form). Against a soft extended Gaussian they keep the canonical form, which needs no decomposition and takes a precision whose rank changes at first order (P25’s limit), which the extended form refuses. HardConstraintFactor’s free energy takes and the null space of by the same rule (hard_constraint.md).

Measured. test/extended_gaussian_ad.jl: beliefs , which turn their flat, hard and Gaussian directions with at constant counts (30 random draws, dimension 2–4); pushforward, preimage, the hard directions’ projector, the canonical part, the product, the support entropy of a product and extended of an improper Gaussian whose null space turns, each through basis-free outputs (, , ): ForwardDiff and Zygote against central differences, absolute and relative (305 checks). With the bases held fixed (the moving terms of _psd_split multiplied by zero) 182 of them fail. Refusals: a precision and a covariance that gain rank at first order, and nested duals; a precision whose null space only turns is accepted. Lenticulum test/hard_gradient.jl: the coupled replay on 12 random trees with a hard relation (some fixing and entirely) against the ForwardDiff gradient of the dense . A belief with no Gaussian direction (only hard or flat ones) is tested on its own: the loop above draws at least one, and the review (LP-043) found Zygote failing there.

What it does not do

  • Density and log-partition. A belief with a flat or hard direction has no density with respect to Lebesgue measure on ; belief_logdensity does not take it. Its entropy is relative to its support (above); with flat directions there is none.
  • Factors. HardConstraintFactor emits it; the message store’s distance and damping take it (Mycelium gaussian.jl); every relation factor takes it as input, and GaussianFactor, GaussianPrior, LinearConstraintFactor and HardConstraintFactor take hard inputs in their free energies. GaussianFactor, NonlinearFactor and LinearConstraintFactor emit it for improper messages; the improper flag of GaussianBelief remains, by decision, for AD, plates and mixtures (Backlog Done 2026-11, 2026-11-11).
  • Gradients where a rank changes. A tangent that changes the number of flat, hard or Gaussian directions at first order has no derivative (the value jumps); ForwardDiff refuses it, Zygote returns the constant-rank derivative without a sign (below). A rank that changes only at second order is not detected. Nested duals (second derivatives) are refused.
  • Device arrays. Host Matrix only.

Difficulties

The constructor first projected and with . When spans everything (every input of a relation uninformative), that leaves a residue of instead of zero, and the relative tolerances took it for a tiny variance: a message that should be flat came out hard. It now projects through an orthonormal basis of the complement of , which is then empty.

Under Zygote, hard_directions of a belief with two or more hard directions and no Gaussian one threw a DimensionMismatch: the split had no nonzero eigenvalue, and the pullback of the broadcasts by the empty could not be projected onto it. _psd_split now returns before them when is empty; at constant rank stays 0 and spans everything, so nothing moves (LP-043 Findings of the review of P4).

The first version of the oracle failed in half the cases with covariances of order . Two faults, both in the test: with two inversions exhaust 256-bit precision, and the projector formed in Float64 carries a rounding of that turns into a variance of in directions that are not flat. The reference now uses , 512 bits, and forms in BigFloat. A test of flatness by failed for the same reason in a different place (the hard directions have precision ); flatness is tested by the reference variance along instead.