Gaussian belief pooling, densities, entropy, damping and public log overlaps live in Mycelium alongside the categorical and mixture rules.
Sources: code:
gaussian.jl,../test/overlap.jlTheory: Gaussian Belief · Belief Algebra · Bethe Free Energy
Implements: the Mycelium belief interface for LenticulumCore.GaussianBelief.
1. The rules
| function | for Gaussians |
|---|---|
combine | addition of canonical parameters: exact, associative, total; keeps the improper flag only beside zero precision (gaussian_belief.md, “Declared improper”) |
combine with a DiracBelief | the Dirac wins (the limit) |
belief_logdensity | closed form; proper beliefs only |
variable_entropy | , may be negative |
belief_distance | max-norm on , defined for improper beliefs too |
can_damp, _damp | convex combination of canonical parameters |
combine_with_overlap | the pooled Gaussian and its log-overlap; proper first argument, second may be improper |
2. History
These methods were in the top-level Lenticulum package with the type, as extensions of
Mycelium’s functions. They moved here when the type moved to LenticulumCore
(gaussian_belief §2), so that the rules sit beside the categorical and mixture rules in
messages. The numerical rules are unchanged; the overlap rule is now public.
Related: messages, free_energy, gaussian_belief
For proper c,d, logZ is log N(μ_c;μ_d,Σ_c+Σ_d). For improper d it is
A(c⊙d)−A(c), with its undefined normalisation constant omitted. That constant
depends only on d and cancels across mixture components. test/overlap.jl
checks the proper formula and differences across two proper first arguments
for a rank-one improper likelihood.
Device diagonal messages (2026-10-27)
Gaussian entropy/density use Core’s precision-logdet helper. The CPU path still
uses logdet of its Cholesky factor; a device Diagonal uses sum(log, diag).
Julia’s generic diagonal logabsdet reduction has no GPU neutral element for dual
pairs, which broke ForwardDiff replay even when the counting coefficient was zero.
Elementwise positive-diagonal algebra avoids this reduction. Exact/pathwise and
terminal score tests with JLArrays compare against CPU and closed-form derivatives
in lib/ImplicitLayers.jl/test/device_arrays.jl. Dense device precision remains
unsupported. See gaussian_belief and gradient_couplings.
Device density/entropy specialisations use only diagonal fields and typed scalar
constants. Mixing a native diagonal matrix-product pullback with direct diag
field reads produced incompatible Zygote cotangents; the elementwise quadratic
form avoids that mix. Generic inv(Diagonal(device)) also scalar-indexes on this
stack: Core’s device moment constructor/covariance accessor instead broadcast
reciprocals. Their derivatives and score estimates are explicitly tested.
Batch C damping precision (2026-10-28)
The GPUArraysCore extension casts damping weights to diagonal-Gaussian or Dirac
payload precision, before broadcasting. A Float64 keyword weight otherwise widens
Float32 device messages. CPU damping arithmetic is unchanged. New realised
flooding/residual replay tests compare Float32/Float64 diagonal Gaussian tree/loop
messages and derivatives against CPU references with scalar indexing disabled.
See test/graph_adjoint_devices.jl and gradient_couplings.
Overlaps without cancellation (2026-10-29, LP-019 items 2 and 6)
combine_with_overlap used to return . Each log-partition holds a
term , and at the difference loses every digit: two
identical gave instead of , and a mixture’s
product weights shifted by became [0.5, 0.5] instead of [0.5312, 0.4688]. Proper
pairs now use the convolution , which
depends only on the difference of the means. For an improper the value is still
(the convention overlap.jl tests),
expanded about . That value is genuinely of size , which is harmless when
the mixture weights are normalised.
Translation invariance holds until the canonical form itself fails: beyond , is recovered only to , which no overlap formula repairs.
The rule needs a proper first argument. _mixture_product now puts a proper Gaussian first,
whichever side of the product it came from, so pooling a mixture with an improper site no
longer depends on the factor insertion order. Tested in test/lp019.jl.
After review (2026-10-30)
Codex’s review of the fix found four gaps, now closed: the proper case called inv on a
Cholesky factor (no Zygote rule) and scalar-indexed diagonal device arrays, so it now goes
through belief_cov and LenticulumCore’s _precision_factor/_solve/_logdet; finite
covariances near floatmax overflowed when added, so is scaled by its
largest diagonal entry; and an improper site far from the origin left log masses of size
, so mixture products re-centre each improper site at one reference point (the first
proper component’s mean on the other side; _overlap(c, d, ref)) and normalise by subtracting
the maximum first. combine_with_overlap(c, d) itself keeps the unshifted convention tested in
overlap.jl.
Second review (2026-10-30): the proper case scales per coordinate,
with
(_scaled_sum, diagonal-preserving, so huge and tiny variances coexist and device/Zygote
paths broadcast); and mixture products re-centre improper sites only when the other side has
exactly one, since the dropped constant differs between sites.
Expectation propagation (2026-11-03)
The Gaussian pieces of the ExpectationPropagationRule() (passing.md):
ep_project(m::MixtureBelief{<:GaussianBelief}): the total mean and covariance , as Lenticulum’smoment_match, but returningnothinginstead of throwing when a component is not proper or the covariance is not positive definite: the rule then skips the update.ep_divide(p, c): , declared improper unless the difference is positive definite. EP sites are often not: on the clutter problem a site’s precision is negative for an outlier the cavity already explains._indefinite(b): a precision with an eigenvalue below ; an EP node skips an update whose factor would receive such a cavity, since the factors take positive semidefinite inputs only.
Products of improper components (2026-11-03)
combine_with_overlap(c, d) used to require a proper c. When both are improper, unnormalised
exponentials as a likelihood mixture carries them, the product is
exact: the exponentials multiply, , and there is no overlap integral to take.
If the sum is proper it is normalised and its mass
moves into ; otherwise it stays an unnormalised
exponential with . With a re-centring point ref (LP-019) the same constant of d is
dropped as in the proper case, evaluated at ref. Two mixtures of likelihoods, the messages
of two data points to one variable, can now be pooled; exact belief propagation on the clutter
problem needs it (test/mixture_rules.jl, which also checks the formula on a 1-d pair and the
improper case).
The normalisation subtracts no log-partitions from each other, but it does compute directly, which is large when the product is far from the origin; the LP-019 care for proper pairs (the convolution form) has no counterpart here.
Extended Gaussians (2026-11-10)
combine takes LenticulumCore.ExtendedGaussians (lib/LenticulumCore.jl/src/extended_gaussian.md):
two of them, or one and a GaussianBelief, combine by the exact product (_conjoin, beside the
type), which stacks hard constraints, adds precisions on what they leave free, and keeps the
directions flat in both. A DiracBelief still dominates, as against a Gaussian, but is checked
first against the hard constraints of the other side: a clamp that violates one is an
ArgumentError, like two unequal clamps. The result of a product with an extended Gaussian is an
extended Gaussian, also when both inputs were representable in canonical form.
The store takes them (2026-11-11). belief_distance compares , and the projector onto
the flat directions (independent of their basis); against a GaussianBelief, DiracBelief or
TrivialBelief it compares the other’s extended form, so a message that changes kind converges.
damp is the geometric mixture LenticulumCore._damp_extended: the canonical rule on Gaussian
directions, gradual on directions flat on one side, the weighted average of two hard values.
Since 2026-11-12 (P1) this holds for a mixed pair too, an extended Gaussian and a GaussianBelief
in either order: the GaussianBelief is damped as its extended form. Before, can_damp had no
method for the pair, and damp returned the new message undamped without a sign, which happens
whenever a likelihood changes form between sweeps (flat while an input is missing, proper once it
arrives). Against a GaussianBelief of dual numbers (ForwardDiff), combine, belief_distance and damp
take the extended side in canonical form when it has no hard directions; with hard directions
there is no canonical form, and since P4 (2026-11-16) the dual side is taken in its extended form,
which moves at constant rank (LenticulumCore extended_gaussian.md, “Derivatives”). Until P4
that pair was an ArgumentError (can_damp was false). Before,
combine threw a MethodError there, inside infer!, on a variable that hears a dual flagged message
and a plain extended one (LP-028 Findings of the review of P1, F1). test/forms_crossing.jl checks combine, belief_distance and damp on
150 random pairs of proper, flagged and zero-precision beliefs, each side in either form, against
the operation with both sides in canonical form (3 003 checks); the damping checks fail against
the code before the fix; and the dual pairs on 60 random instances (1 080 checks). Not checked: the distance near the rank threshold. A message whose
precision crosses it between sweeps changes form, and its distance in extended form then counts
the change of the flat subspace (up to 1) and the large variance, where the canonical distance
would be small. This is expected from the definitions, not measured on a run.
HardConstraintFactor (Lenticulum) emits extended Gaussians under any schedule. Tests:
test/runtests.jl, “combine with extended Gaussians” and “the message store takes extended
Gaussians”.
Entropy (2026-11-11). variable_entropy of an extended Gaussian is
LenticulumCore.support_entropy: the entropy relative to Lebesgue measure on the belief’s support,
0 for a point, as for a DiracBelief. A variable that a hard constraint fixes in some directions
therefore has a finite counting term, and flat directions (an improper marginal) throw. On trees
with a hard relation the Bethe free energy is then (Lenticulum test/hard_free_energy.jl;
the convention is in lib/LenticulumCore.jl/src/extended_gaussian.md, “Entropy”).
Mixture reduction (2026-11-12, P17)
reduce_mixture(m; max_components, prune) and reduce_message(::MixtureBelief{<:GaussianBelief}, M),
the message of a MixtureReduction(M) node, are defined here. They were in Lenticulum’s beliefs.jl,
where the reduce_message method was type piracy (Mycelium’s function on LenticulumCore’s types),
and Mycelium without Lenticulum refused to reduce a Gaussian mixture. The code is unchanged; Lenticulum
re-exports reduce_mixture. The theory (Runnalls’ bound on the KL divergence of a merge, which keeps the
total mean and covariance) and the refusal of improper components over the budget are in Lenticulum’s
beliefs.md, “The reduction of a message”.
Tested in test/mixture_reduction.jl with Mycelium and LenticulumCore only: on 30 random mixtures of
2–6 components, every budget from 1 to K + 1 gives the mixture unchanged within the budget, the
moment-matched Gaussian at one, and otherwise the requested number of components with the total mean
and covariance kept to . Mistral compared the moved functions with the code before the move on
40 random mixtures and found them identical (LP-029 Findings of the review of P17 and P18).