OverlayFactor: several different factors on the same variables, each with its own parameters, as one node. Unnormalised it is their product, equal to a cluster node of the members. Withnormalise = (:y, …)it is, for each joint outcome of the members’ switches, the product normalised as a density ofy: a product of Gaussian experts, and a mixture of experts with switches. A learned relation on data cannot collapse then.
Sources: original to this repository (RQ-008 Overlays, Mixtures and Tropical Limits, the Feature Backlog’s “mirrored factors”). code:
overlay.jl; tests: Lenticulumtest/overlay.jl
1. What it is
OverlayFactor(:a => f₁, :b => f₂, …) holds members stated by a cluster_form. Its channels are the
union of theirs; members with a channel of one name share that variable, switch channels included.
Its parameters are a NamedTuple keyed by the member names, each member with its own.
Unnormalised, the energy is . In cluster-form terms the overlay is one form:
- the members’ residuals are stacked;
- their noises form a block-diagonal ;
- its outcome is the joint outcome of all the members’ switch channels, column-major in their order;
- its log-weights are the sum of the members’ for that joint outcome.
If any member’s coefficients or noise depend on its outcome, so do the overlay’s.
Normalised in channels , for each joint outcome :
with , , the stacked form and the product of the members’ weights. It is outcome by outcome: the -quadratic of the summed energies, completed. So members that do not involve cancel. must be invertible (the members must determine ), or the form refuses. A common scaling of the coefficients and the noise leaves and the mean unchanged.
Outside a cluster its messages and free energy are those of a one-factor cluster (as_cluster).
Inside one it is its cluster form.
2. The normaliser and LP-022
A constraint factor with every coefficient learned collapses on data (A = 0). Normalised in its
data channel it is a conditional density there: the normalised relation is
, so its coefficients and its scale are the conditional mean and the
variance. learns_unnormalised is false for a normalised overlay, so no warning fires.
test/overlay.jl checks this with one fully learned LinearConstraintFactor on 100 points of
:
- is refused (singular );
- at the maximum-likelihood point (least squares, its residual variance) the free energy equals to ;
- it is lower there than at perturbed parameters;
- the gradient vanishes there.
3. Normalising outcome by outcome
With switches the normalisation is per joint outcome: , a mixture of experts whose weights do not depend on . The exact normalisation of the whole product, , gates the outcomes by . It is a different model and not a cluster form, because of the division by a sum. Without switches the two agree.
4. Checked by
test/overlay.jl:
- Unnormalised overlay of a linear constraint and a switch: equals a cluster node of the same members as separate nodes (free energy and outcome posterior to ). Its message to the outcome outside a cluster matches the closed form.
- Normalised:
- two linear experts give the precision-weighted Gaussian;
- a linear expert and a switch give the mixture of experts ;
- a prior on in the overlay cancels;
- all to .
- The LP-022 case of §2.
5. What it does not do
- Members without a
cluster_form(a nonlinear energy, a convex ) are not supported. They would need Laplace or EP messages (RQ-008, leads). - No closed-form M-step for the overlay (
m_step); EM keeps its parameters, and gradients learn them. - Only per-outcome normalisation with switches (§3).
- No batching over plates beyond what its cluster form gets as a cluster node.