design derivation

A factor learned by the free energy can satisfy its data trivially: a relation that holds everywhere (a constant or zero residual), a noise that shrinks with its coefficients, an unnormalised potential that grows without bound. What prevents it is always the same thing: the factor is −log of a density normalised in its data channels, or it is trained by comparison. This note goes through every learnable factor family, says which of them are safe, which are guarded, and lists the options. Since 2026-11-05 every collapse found is at least warned.

Sources: LeCun, Chopra, Hadsell, Ranzato & Huang, A Tutorial on Energy-Based Learning, 2006, §2 (loss functionals) and §5 (the energy loss and flat energy surfaces); original analysis of this repository’s factors. code: lib/Lenticulum.jl/src/constraint.jl (_coeffs, output), lib/Lenticulum.jl/src/regime.jl, lib/Lenticulum.jl/src/m_steps.jl, lib/Mycelium.jl/src/free_energy.jl (learns_unnormalised), lib/AlgebraicFactors.jl/src/toric.jl, lib/AlgebraicFactors.jl/src/polynomial.jl, lib/ImplicitLayers.jl/src/deq.jl; tests: test/lp022.jl, lib/AlgebraicFactors.jl/test/lp022.jl, lib/Lenticulum.jl/test/learnable_noise.jl

Bibliography: LeCun et al. 2006

Theory (CT-ML wiki): Variational Free Energy

1. The problem

A Mycelium graph scores its parameters by the free energy . Data enter as point messages (DataFactor), latent variables are integrated out, and

Nothing normalises over the observed variables: there is no partition function over the data anywhere in the graph. So is only if every factor that touches data is itself normalised in those channels. Otherwise is what LeCun et al. call the energy loss (§2): it pushes the energy of the observed configuration down and nothing else up, and a model can satisfy it by making the energy small everywhere. Energy-Based Learning §3 has the general argument; this note is about the concrete factors.

Three ways to collapse, each met in the code:

  1. A constant relation. A residual that is zero for every (all coefficients 0, or a map that is the identity on its own output) puts every point on the relation.
  2. Scale. If the coefficients and the noise are both learned, and multiply the factor by : the free energy falls without bound as .
  3. An unnormalised potential. without a log partition falls without bound along the observed state’s features.

2. When a factor is safe

A learned factor is safe on data if, for every value of its parameters, is a normalised density (or probability) in its data channels, given the others. Then is of a likelihood, which is bounded below for any fixed data set that is not fitted exactly (a perfect fit of finitely many points is overfitting, not collapse; see the Gaussian row below). There are five ways to get there, and the factors below use each of them:

mechanismwhat it fixesexample
a fixed coefficient on the data channelthe change of variables GaussianFactor (), output = :y
a normalised noise modelthe volume is paidlearned noise in GaussianFactor, RegimeFactor
a log partition is subtractedToricFactor(...; normalised = true)
a constrained parameter setthe trivial parameter is excludedPolynomialFactor on the Grassmannian
comparison against somethinga contrastive term pushes energy upRatioFactor (classification against ), denoising

3. The factors

factorlearnednormalised in its data?can it collapse?guard
GaussianFactor, , optionally yes: , no; with learned , an exact fit drives noise prior or floor (lib/Lenticulum.jl/test/learnable_noise.jl)
GaussianPriorthe noise, optionallyyesas aboveas above
NonlinearFactor, LikelihoodFactornothing (islearnable = false)yesno—
LinearConstraintFactor, only in the residualyes: (1)output = :y; a warning; m_step refuses
SwitchingConstraintFactor, , only in the residualyes (1)as above
RegimeFactor, , , optionally only in the residualyes (1), and (2) with learned noiseas above
cluster nodesthe clustered factors’as their factorsas their factorsthe factors’ guards; tied parameterisations must keep their normalisation (below)
TableFactornothing—no—
ToricFactoronly with normalised = trueyes (3) when unnormalised (the default)a warning (since 2026-11-04)
PolynomialFactorno (Sampson energy)not to ; can interpolateGrassmann parameter; no graph free energy
DEQFactorthe cell no: yes (1) with observed: a warning when is observed (since 2026-11-05)
NeuralODEFactorthe fieldno, but ‘s coefficient is fixedno: is a regression—
LuxFactorthe layerresidual = true: no (a fixed-noise regression); default energy 0—
GeneratorFactorenergy 0nothing to collapse; no signal of its ownlearned by comparison
RatioFactorthe discriminatorby comparisonno flat solution; it estimates contrastive by construction
DiffusionFactorthe noise predictortrained offline by the denoising lossno: costs the denoising objective; its graph free energy is not a training objective

Constraint factors (LinearConstraintFactor, SwitchingConstraintFactor, RegimeFactor)

Their energy is with : normalised over , not over any channel. They are relations, and every coefficient is a parameter by default. Measured in test/lp022.jl (200 points of , ): least squares has , the constant relation has , and at least squares the gradient in is , so descent shrinks the relation. With learned noise (RegimeFactor(...; learn_noise = true)), halving and the noise’s scale lowers by exactly (2).

Guards (LP-022 Constraint factors are normalised in their residual, not their channels):

  • output = :y fixes the coefficient of the data channel (identity, or an invertible output_coeff) and removes it from the parameters. The factor is then the normalised conditional density of given the others, up to , and fits by gradient or EM reach least squares (tested).
  • Freezing a coefficient (freeze = (:A,) or (:A, :y)) does the same by hand.
  • free_energy_gradient warns once when such a factor, learnable and unfrozen and with no output, touches a DataFactor variable (Mycelium.learns_unnormalised).
  • The closed-form m_step refuses a factor whose every coefficient is learned, because its minimiser is .

Cluster nodes inherit this. A cluster’s free energy is only when its factors are normalised. A parameterisation that ties several factors must keep the normalisation itself. In lib/Lenticulum.jl/examples/structure_clusters.jl the switches’ weights are normalised per cluster so that the tables’ makes the joint a mixture. An unconstrained per-factor M-step would break that, and its free energy would no longer be . The example’s EM therefore uses its own constrained M-step.

Toric factors

ToricFactor is the log-linear potential , unnormalised by default. On observed states its energy has no lower bound. Measured in lib/AlgebraicFactors.jl/test/lp022.jl (two binary variables observed, ): for . With normalised = true the log partition is included and ; it is 0.0 there, the single observation fitted perfectly. The gradient shows the same thing (toric.md): , and the contrastive second term is exactly the partition’s. Unnormalised toric factors are right as potentials between latent variables, and wrong as learned models of data. Since 2026-11-04 learns_unnormalised(::ToricFactor) makes free_energy_gradient warn for them.

Polynomial factors

PolynomialFactor scores by the Sampson distance with . That is scale invariant in , and lives on the Grassmannian (orthonormal rows), so the zero polynomial is not a parameter (polynomial.md). It is fitted by fit_sampson, and no graph free energy is supplied, so it is not trained by . What remains is the ordinary algebraic overfitting: with more monomials than points, a nonzero polynomial can vanish on all of them. That is a choice of degree, which fit_nullspace’s nullity diagnoses. It is not a collapse of the parameterisation.

Implicit layers

  • DEQFactor: with and no normalising term (deq.md §4.3). With and both observed and the energy used as a training objective (free_energy_gradient scores fixed incoming endpoints, deq.md §4.5), a cell with puts every pair on the relation. It is a collapse of kind (1), because the coefficient of is not fixed: appears inside . Measured in lib/ImplicitLayers.jl/test/lp022.jl (50 pairs of , a linear cell ): the generating relation has , the identity cell (, ) , and at the generating relation the gradient in is , so descent moves towards 1. Since 2026-11-05 learns_unnormalised(::DEQFactor, observed) makes free_energy_gradient warn when the state channel is observed (the input alone is the usual case and does not warn).
  • NeuralODEFactor: (neuralode.md §4.4). The coefficient of is fixed, so this is a regression with unit noise, safe in the sense of §2.
  • LuxFactor: energy zero by default, so it delegates to other factors and has nothing to collapse. With residual = true it is , a fixed-noise regression.

Comparison-trained families

  • GeneratorFactor: energy zero (generator.md §4.2). It cannot learn from its own free energy and is trained by comparison, as Mohamed & Lakshminarayanan argue implicit models must be (Implicit Generative Models).
  • RatioFactor: a classifier between and . The contrastive term is the other class, so its logit converges to (ratio.md §1), which is noise-contrastive estimation (Training Energy-Based Models). The generator side can still fail in the usual GAN ways (mode collapse); nothing here measures that.
  • DiffusionFactor and the implicit relations: the predictor is trained offline by the denoising loss , whose target is the injected noise. A constant predictor costs , so there is no flat solution. The graph free energy of a diffusion factor is Monte Carlo and not a training objective (factor.md §5.3). The relation it defines (the stable roots of the smoothed score) has a different defect: those roots are modes of , not points of the data manifold (LP-023 Findings of the external reviews of 2026-11-04, “implicit-relation calibration”).

4. Options for a factor that is not safe

In order of preference:

  1. Make it a conditional density. Normalise it in its data channel (OverlayFactor(:f => f; normalise = :y), since 2026-11-05: every coefficient stays learnable, and the relation becomes the conditional Gaussian of ; lib/Mycelium.jl/src/overlay.md §2), fix the coefficient of the data channel (output), or write the factor as with outside . For a DEQ that means training through a downstream loss on the solved state, or a residual with not inside .
  2. Pay the volume. Learn the noise together with its , and guard against an exact fit with a noise prior or floor. For an invertible map (a flow) the log-Jacobian is the volume term: would make a DEQ a density in . That isn’t implemented.
  3. Normalise over the states. Use a log partition where it is computable (ToricFactor(...; normalised = true), categorical factors).
  4. Constrain the parameters away from the trivial set: a manifold (the Grassmannian), a fixed norm, a frozen block (freeze).
  5. Train by comparison: classification against noise or a reference (RatioFactor, NCE), denoising, or a margin loss with negatives (LeCun et al.’s generalised margin losses). The framework has no slot for a loss functional (Energy-Based Learning §3), so this happens outside the free energy.

5. What the code enforces today

checkwherecovers
warning in free_energy_gradientMycelium.learns_unnormalised, _unnormalised_nodesthe three constraint factors without output; ToricFactor without normalised (any observed channel)
refusal in the closed-form M-steplib/Lenticulum.jl/src/m_steps.jlconstraint factors with every coefficient learned
outputthe three constraint factorsmakes them conditional densities
parameter validationvalidate_parametersan output coefficient passed as a parameter
warning in free_energy_gradient, by channellearns_unnormalised(f, observed)DEQFactor with its state observed

The warning fires only for data that reach the factor through a DataFactor. Pinned variables and point messages from other factors are not detected.

Energy-Based Learning · Training Energy-Based Models · Bethe Free Energy · Switching Constraint Factors · LP-022 Constraint factors are normalised in their residual, not their channels