ExtendedGaussian(μ, Σ, D)(extended_gaussian.jl): a Gaussian made flat along a subspace, the belief type for linear relations. Three kinds of direction in one object: flat (D, a likelihood that says nothing there), hard (null(Σ)outsideD, a linear constraint imposed exactly) and Gaussian. Pushforward through affine maps and the product are exact;GaussianBelief,DiracBeliefandTrivialBeliefare special cases and convert to it.
Sources: Stein & Samuelson, A Category for unifying Gaussian Probability and Nondeterminism, CALCO 2023 (arXiv:2204.14024: subspaces admitted as generalised uniform distributions); Stein & Samuelson, A Categorical Treatment of Open Linear Systems, LMCS 21(3), 2025 (arXiv:2403.03934: extended Gaussian distributions as Willems’ Gaussian linear systems). code:
extended_gaussian.jl; Mycelium’scombinemethods inlib/Mycelium.jl/src/gaussian.jl; first consumer_improper_splitinlib/Lenticulum.jl/src/gaussian.jl. tests:test/extended_gaussian.jl, Myceliumtest/runtests.jl(“combine with extended Gaussians”).Bibliography: Stein & Samuelson 2023 · Stein & Samuelson 2025
Theory (CT-ML wiki): Gaussian Relations
Theory
An extended Gaussian on is the law of with and “uniform” on the subspace : a Gaussian on the quotient . It is the limit of the proper Gaussians with covariance as ( the orthogonal projector onto ). Allowing to be singular adds the other limit: along there is no variance, and the belief imposes exactly ( a basis of those hard directions).
Every belief the Gaussian code handles is one of these:
| belief | ||
|---|---|---|
proper GaussianBelief | , full rank | |
improper GaussianBelief (a likelihood flat along ) | ||
DiracBelief(v) | (every direction hard) | |
TrivialBelief() | ||
| a hard linear constraint , nothing else | on |
The canonical form covers the first two and the zero precision; it cannot hold a hard direction (infinite precision). The moment form covers hard directions and not flat ones. The extended form holds both, which is what a factor graph of linear relations needs: a relation that constrains of coordinates sends a message flat in directions, and a noise-free relation imposes its directions exactly.
Pushforward. For with independent, the law of is . The image of a uniform law on is uniform on , so this is exact for every matrix , rectangular or rank-deficient. Marginalising onto coordinates is the pushforward through the selection matrix.
Preimage. For a belief about , the likelihood is again an
extended Gaussian on : hard where would have to lie in a hard direction of
(), Gaussian with precision and information
, flat along and the preimage of ‘s flat directions. With
the pushforward it gives the message of every linear relation (HardConstraintFactor,
lib/Lenticulum.jl/src/hard_constraint.md).
Product. Two extended Gaussians as potentials on : each is supported on an affine subspace (its hard constraints) and, there, is , constant along . The product is supported on the intersection: stack the constraints into , take a particular solution (refusing the product when none exists: contradictory constraints) and an orthonormal basis of , and write . The precisions add, , , and in the quadratic part is , . Along the product is flat; on the rest it is Gaussian with mean and covariance . Mapped back: , , . No component of can lie along : a vector there has (both are positive semidefinite), so . The product of two extended Gaussians is therefore always one, or a contradiction.
Implementation
- Normalised representative. The constructor orthonormalises (an SVD, dependent columns dropped) and projects and onto , since nothing along is observable. Two representatives of one law then agree up to the basis of ; the tests compare .
- Tolerances. An eigenvalue or singular value below counts as zero, the
rule
_split_momentsused before. So a direction with variance below about of the largest is hard: a deliberate choice, since a covariance that small is a constraint in all but name, and treating it as Gaussian is what made the rank-deficient messages of LP-024 wrong. Contradictory constraints are those whose least-squares residual exceeds times the scale of the means. - Conversion.
extended(::GaussianBelief)splits by the eigendecomposition of (and refuses with a negative eigenvalue beyond rounding, or along : along a flat direction is no limit of Gaussians).GaussianBelief(::ExtendedGaussian)goes back when there is no hard direction, declaring the result improper when . - One routine for products and preimages.
_from_potential(C, c, η, Λ)builds the extended Gaussian of ; the product andpreimageboth reduce to it._independent_joinstacks independent beliefs (means, block-diagonal covariances and flat directions). - Damping (Mycelium’s
damp) is the geometric mixture , the rule canonical damping of aGaussianBelieffollows:_conjoinwith weights, precisions scaled. Hard constraints of the two sides on one direction are not a contradiction there: withstrict = falsetheir values are the weighted least-squares fit, the limit of precisions and , so a hard value moves by the damping weight like any other. - Moments.
belief_meanandbelief_covreturn and when the belief is flat nowhere (hard directions have variance zero); along a flat direction there is no mean. - Where the product lives. The algebra is here, beside the type (
_conjoin); Mycelium’scombinemethods call it, for two extended Gaussians or one and aGaussianBelief. ADiracBeliefstill dominates, as it does against a Gaussian, but is first checked against the other side’s hard constraints. - First consumer.
_improper_splitin Lenticulum’sgaussian.jl, which splits a partly flat input for the constraint, switching, nonlinear and Gaussian factors, is nowextended(b). Its outputs are the same up to the basis of the flat directions, which the callers use only through its span. The split itself is_flat_split(η, Λ), which returns the parts without building the type: Lenticulum’s split of a flagged input with a dual and a precision of plain values calls it directly, and the mean carries the duals (P21). Its range check reads ‘s values (stop_gradient):normof duals with value exactly zero isNaN.
Entropy (2026-11-11)
A belief with hard directions has no density with respect to Lebesgue measure on , but
it has one on its support, the affine subspace . support_entropy(b)
is the entropy relative to Lebesgue measure there, in orthonormal coordinates:
over the nonzero eigenvalues of . It is the finite part of the entropy of
, whose divergent part is .
A point gets , the convention Bethe Free Energy already uses for clamped variables. With flat
directions the belief is improper and has no entropy; support_entropy throws.
This is a convention, and the Bethe sum is only meaningful if every term follows it: each
belief’s entropy relative to its own support. A factor with a smooth density whose input has hard
directions works in the input’s coordinates along the other directions. _hard_reduction(b)
returns , with the point fixed along the hard directions, an orthonormal
basis of the rest, and the Gaussian part in in canonical form, ,
zero along the flat directions. The hard relation’s own term is in HardConstraintFactor’s note.
On trees the result is (Lenticulum test/hard_free_energy.jl), which is what makes the
convention the right one there.
Measured
test/extended_gaussian.jl:
- The product against the limit of proper Gaussians, 135 random pairs (dimension 2–5, 0 to flat and up to hard directions between the two, in random bases). The reference is the product of the Gaussians with covariance , , , in 512-bit arithmetic. Off the product’s flat directions the means and covariances agree to ; along them the reference variance exceeds .
- Laws: commutativity, associativity, the flat belief as unit, pushforward composing
(
pushforward(pushforward(b, M₁), M₂) = pushforward(b, M₂M₁)),marginaliseas a pushforward. support_entropyagainst the limit, 60 random beliefs (dimension 1–5, 0 to hard directions): the entropy of minus at in 512-bit arithmetic, with formed there from the generator’s factors (exactly of rank ), agrees to ._hard_reduction, 40 random beliefs with flat and hard directions: orthonormal and orthogonal to the hard directions, the fixed point along them, the precision , zero along the flat directions.- Conversions and refusals: round trips with
GaussianBelief, the clamp, contradictory constraints, outside , a covariance that is not positive semidefinite.
Derivatives (P4, 2026-11-16)
The type holds any Real (ExtendedGaussian{T<:Real}), so a ForwardDiff dual or a Zygote-tracked
float passes through every operation. Before, it held only floats, and the coupled replay of a
graph with a HardConstraintFactor failed under both (ForwardDiff at the type bound, Zygote at
_independent_join, which wrote into zero arrays and took no tuple).
The rule. Every subspace (a basis of the flat directions, of their complement, of the hard
directions, of the range of a covariance or precision, of the solution set of stacked constraints)
comes from a decomposition of the values, and its rank is decided there by _eig_tol. The
derivative is the one at constant rank. With the value of a matrix and
(zero in value, the tangent in derivative), the bases move
only into each other:
- a positive semidefinite with range (eigenvalues ) and null space :
, , and
, whose derivative is the constant-rank one,
(
_psd_split,_split_pinv); - the column space of : (
_orth); the complement of the column space of : (_complement); - the least-squares solution and null space of stacked constraints , with at the
value: ,
(
_from_potential); - the support entropy uses , whose first-order change is that of the nonzero eigenvalues.
No eigenvector or singular vector is differentiated, so repeated eigenvalues are no difficulty;
these are the formulas of P25 (LenticulumForwardDiffExt) and P26, written once for both ADs: the
decompositions are of values (_value_eigen, _value_svd, marked non-differentiable for
ChainRules), and carries the derivative through stop_gradient, which strips a dual and has
a zero pullback. With floats is exactly zero and every result is the decomposition’s.
Refusals. A tangent with , and the left and right null spaces at
the value, changes the rank at first order: the value jumps there and has no derivative.
_check_constant_rank (LenticulumCoreForwardDiffExt) refuses it under ForwardDiff, with the
tolerance per partial. A Zygote cotangent cannot see
the tangent, so under Zygote the constant-rank derivative is returned without a check.
Where it is used. Mycelium’s combine, damp and belief_distance of a GaussianBelief of
duals and an extended Gaussian with hard directions now take the dual side’s extended form
(before: an ArgumentError, there being no canonical form). Against a soft extended Gaussian they
keep the canonical form, which needs no decomposition and takes a precision whose rank changes at
first order (P25’s limit), which the extended form refuses. HardConstraintFactor’s free energy
takes and the null space of by
the same rule (hard_constraint.md).
Measured. test/extended_gaussian_ad.jl: beliefs ,
which turn their flat, hard and Gaussian directions with at constant counts (30 random draws,
dimension 2–4); pushforward, preimage, the hard directions’ projector, the canonical part, the
product, the support entropy of a product and extended of an improper Gaussian whose null space
turns, each through basis-free outputs (, , ): ForwardDiff and Zygote against
central differences, absolute and relative (305 checks). With the bases held fixed
(the moving terms of _psd_split multiplied by zero) 182 of them fail. Refusals: a precision and a
covariance that gain rank at first order, and nested duals; a precision whose null space only turns
is accepted. Lenticulum test/hard_gradient.jl: the coupled replay on 12 random trees with a hard
relation (some fixing and entirely) against the ForwardDiff gradient of the dense .
A belief with no Gaussian direction (only hard or flat ones) is tested on its own: the loop above
draws at least one, and the review (LP-043) found Zygote failing there.
What it does not do
- Density and log-partition. A belief with a flat or hard direction has no density with
respect to Lebesgue measure on ;
belief_logdensitydoes not take it. Its entropy is relative to its support (above); with flat directions there is none. - Factors.
HardConstraintFactoremits it; the message store’s distance and damping take it (Myceliumgaussian.jl); every relation factor takes it as input, andGaussianFactor,GaussianPrior,LinearConstraintFactorandHardConstraintFactortake hard inputs in their free energies.GaussianFactor,NonlinearFactorandLinearConstraintFactoremit it for improper messages; theimproperflag ofGaussianBeliefremains, by decision, for AD, plates and mixtures (Backlog Done 2026-11, 2026-11-11). - Gradients where a rank changes. A tangent that changes the number of flat, hard or Gaussian directions at first order has no derivative (the value jumps); ForwardDiff refuses it, Zygote returns the constant-rank derivative without a sign (below). A rank that changes only at second order is not detected. Nested duals (second derivatives) are refused.
- Device arrays. Host
Matrixonly.
Difficulties
The constructor first projected and with . When spans everything (every input of a relation uninformative), that leaves a residue of instead of zero, and the relative tolerances took it for a tiny variance: a message that should be flat came out hard. It now projects through an orthonormal basis of the complement of , which is then empty.
Under Zygote, hard_directions of a belief with two or more hard directions and no Gaussian one
threw a DimensionMismatch: the split had no nonzero eigenvalue, and the pullback of the broadcasts
by the empty could not be projected onto it. _psd_split now returns before them when
is empty; at constant rank stays 0 and spans everything, so nothing moves
(LP-043 Findings of the review of P4).
The first version of the oracle failed in half the cases with covariances of order . Two
faults, both in the test: with two inversions exhaust 256-bit precision,
and the projector formed in Float64 carries a rounding of that turns into a
variance of in directions that are not flat. The reference now uses , 512 bits,
and forms in BigFloat. A test of flatness by failed for the same
reason in a different place (the hard directions have precision ); flatness is tested
by the reference variance along instead.