implementation

Cluster nodes: cluster(g, name, factors) replaces a set of factor nodes by one node that runs exact inference inside (enumeration of the discrete states, Gaussian elimination of the continuous ones) and looks like one factor outside, with exact messages and a closed-form local free energy. A whole switching chain in one cluster has the free energy to ; three switches in one cluster send the messages of E7’s hand-built block factor.

Sources: Yedidia, Freeman & Weiss, Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms, IEEE Trans. Inf. Theory 51(7), 2005 (regions; here a region is solved exactly); de Jong, The Diffuse Kalman Filter, Ann. Statist. 19(2), 1991 (the diffuse likelihood: flat directions). code: cluster.jl; Lenticulum src/switching.jl (cluster_form for the constraint factors and GaussianPrior); tests: ../test/cluster.jl, Lenticulum test/cluster.jl

Bibliography: Yedidia et al. 2005 · de Jong 1991

Theory (CT-ML wiki): Variational Free Energy

Implements: L-SPEC-009 Cluster Nodes, Extending Mycelium §5.

What a cluster computes

Every factor in a cluster states itself through cluster_form(factor, ps) as a conditionally linear-Gaussian term: a residual with noise and weight , the column read from one discrete channel or fixed, so that . A discrete-only factor (a categorical prior) has no rows. Mycelium implements the form for PriorFactor (categorical or proper Gaussian belief), Lenticulum for LinearConstraintFactor, SwitchingConstraintFactor and GaussianPrior.

For a joint assignment of the cluster’s discrete variables, the product of the factors and the incoming messages is over the free continuous variables, with the same for every assignment (the forms share and across outcomes), , and collecting the quadratic constants, the normalisers, the log-weights and the incoming discrete log-probabilities. Points (data, pins) are substituted into the constants; a Gaussian-mixture input is collapsed to its moments.

  • Message to a continuous channel . Eliminating the other continuous variables (a Cholesky of , shared) leaves per assignment with and the mass . The message is the mixture over the assignments, normalised components when is positive definite, unnormalised exponentials otherwise, as the switching factor’s.
  • Message to a discrete channel. .
  • Local free energy. The belief is, given , the Gaussian , and . So is closed form, and sums each factor’s expected quadratic and its normaliser. Graded as (fit, complexity, negentropy), as the switching factor’s local free energy, so a cluster of switches and its block factor agree term by term.
  • Gradient. factor_belief freezes , the means per assignment and the covariance; belief_free_energy re-evaluates every factor’s form at ps (LP-020). At an exact belief this is the gradient of . The bookkeeping of which outcome and which channels each factor reads is _cluster_layout, non-differentiable.

cluster(g, name, factors) builds the new graph: the variables touched only by the clustered factors become internal, the others are the node’s channels (named after the variables, on bidirectional edges); a variable is discrete when a clustered factor switches on it (cluster_switch). The node’s parameters are a NamedTuple keyed by the clustered factors’ parameter keys, so tied factors inside keep one entry (cluster_parameters rearranges a parameter tree). cluster_marginal reads the marginal of any cluster variable, internal ones included.

Tests

Lenticulum test/cluster.jl:

  • the switching chain of test/mixture_rules.jl () with every factor but the data in one cluster: the free energy equals by enumeration to (relative), every ‘s marginal mean and variance match the enumerated posterior to , and the gradient in the tied offsets matches central differences of (reference differences , ForwardDiff and Zygote );
  • E7’s additive wiring (, one two-outcome switch per ): the cluster of the three switches sends the message of the hand-built 8-outcome block factor (log-densities to ), the same loopy schedule gives the same marginals on both graphs to , and the two nodes’ local free energies agree term by term to ; a cluster of one switch is that switch;
  • the same wiring with the sum and the switches in one cluster: the plain graph has no free energy (mixture marginals at the ), the clustered one has to ;
  • refusals: a factor without a form, a missing name, a taken name, a key tied outside, a plate.

Mycelium test/cluster.jl: priors only, the message is the product of the Gaussian priors, an internal categorical keeps its prior, and the refusals.

Grouping by overlap and several clusters at once (2026-11-03)

cluster(g, [name₁ => factors₁, name₂ => factors₂, …]; params) makes several cluster nodes in one call. With params they read one parameter entry, tied: one cluster per data point of one model (examples/structure_edits.jl). Without it a parameter key shared between groups is refused, since each node would hold its own copy. cluster_parameters(gc, ps) converts for every node.

overlap_groups(C, maxsize) is E7’s greedy grouping (Composing Stochastic Factors): start from singletons, merge the two groups whose largest pairwise overlap is largest while the merged group fits and the overlap is positive. Lenticulum’s switch_overlaps(g, ps, sum, switches) gives the overlaps of switching factors feeding one linear factor: each switch’s offset directions mapped through the sum’s coefficients, in the metric of the sum’s total noise . For binary switches this is E7’s , : test/cluster.jl compares both to and the groups with E7’s own function, copied as the oracle, for group sizes 2, 3 and 4; two planted overlapping pairs end up in the same groups.

Whether it helps, bench/cluster_overlap.jl (, eight binary switches in four overlapping pairs, interleaved so that index order splits them, 100 points from the model, 50 loopy sweeps), the mean over points of the mean against the exact posterior (every factor in one cluster):

E-stepmean max
one switching factor per switch0.09220.2376
clusters of the overlap groups, the switches only0.09220.2376
E7’s blocks of 4 in index order (partial sum + switches per cluster)0.09170.2433
E7’s blocks of 4 grouped by overlap0.03090.1446

Grouping by overlap cuts the error to a third, as in E7, but only with E7’s wiring: each block’s switches feed a partial sum inside the block’s cluster, and the blocks’ sums feed the sum node. Clustering the switches alone changes nothing, to four digits: they meet only at the sum node, which collapses each ‘s mixture to its moments separately, so the cluster’s joint over its switches never reaches the coupling. Blocks that ignore overlap help as little.

Discrete tables, joint outcomes and impossible assignments (2026-11-03)

A ClusterForm’s switch may be a tuple of discrete channels; the outcome is then their joint state, in column-major order (the first channel fastest, LinearIndices). TableFactor(dims, logtable) uses this: a fixed factor over discrete variables, a discrete-only term () inside a cluster. Outside one it sends exact belief-propagation messages (the table summed against the other channels’ categorical messages, or sliced at an observed state); it has no local free energy there.

Its entries may be , so a table can tie discrete variables together. That is how examples/structure_clusters.jl writes E3b’s switch over sums: each switch reads an outcome with states, and a table over allows only the of cluster . Enumerating every joint state would cost assignments where are possible, so _cluster_joint now adds every factor’s log-weights first and drops the assignments of weight zero before the Gaussian part (the precisions and solves). The beliefs keep the surviving assignments only. Since 2026-11-04 the enumeration itself is pruned (_enumerate_assignments): the assignments are built one discrete variable at a time, a factor’s log-weights are added once all its switch channels are assigned, and a partial assignment of weight zero is dropped there. The 2^16 cap now counts the possible assignments, not the full product: 18 binary variables tied equal by tables (2^18 joint states, 2 possible) form one cluster (test/cluster.jl, “pruned enumeration”; before, the cap refused it). It does not speed up the plated example: at , 2 048 joint states of which 32 are possible, a fitting step took 0.155 s against 0.158 s before (a scratch timing), since the batched path builds one joint per plate group.

A cluster’s message merges equal components: assignments the target does not see (an internal discrete variable that does not move it) gave repeated components with split weights; they are now one component with the summed weight, and a message whose components are all equal is one Gaussian (test/cluster.jl, Mycelium’s side). In belief_free_energy a log-weight met a weight of exactly 0 and gave 0 · (-Inf) = NaN; such entries are set to 0 before the product (_finite_logw), which is the convention, and no AD backend sees the infinity.

Tests (test/cluster.jl): three categorical variables with two tables (one entry each) in one cluster give and the marginals of enumeration to , and belief propagation through the tables (the graph is a tree) the same marginals; the observed-state message and the column order. In Lenticulum, the switch over sums with , in one cluster gives to (a scratch check), and the example checks it at on 200 points () along with the gradient against central differences (examples/structure_clusters.md).

Cluster nodes over a plate (2026-11-04)

cluster(g, plate, name, factors; params) clusters the template factors factors of every replica of plate into one tied node per replica, and the graph keeps the plate. Inside each node the variables carry their template names (:z, not z[3]), so every replica has the same channels, which a plate group needs. as_plate(g, name, n) (plates.jl) records the nodes named x[i] of an unplated graph as a plate; structure edits refuse plates, so an edited graph gets its plate back this way.

Tied clusters differ only in what their replica’s messages bring. When every replica has the same factors (compared field by field) and its inputs the same shape (which channels are points, the input precisions, the discrete inputs), replica ‘s joint is replica 1’s with and : collects the Gaussian inputs’ and the points’ shifts , the terms . So factor_messages builds one joint, its precision’s factorisations and the surviving assignments once (cluster_batch.jl), and factor_beliefs returns FrozenClusterBeliefs, whose belief_free_energies evaluates all replicas as a few stacked matrix products: what the parameter gradient differentiates. Replicas that differ fall back to the per-node methods.

Tests (test/cluster.jl, “Cluster nodes over a plate”): the switch over sums (, a table) on 12 points, plated against the same graph without the plate record: free energy equal to and to by enumeration, the batched beliefs taken, the gradient equal to with ForwardDiff and Zygote, batched messages equal to one by one, the fallback chosen when one replica’s input is a Gaussian instead of a point; as_plate’s refusals. On 50 points at a scratch timing gave sweeps plus free energy 3.7 times and the gradient 6 times faster than the unplated clusters, at equal values (, ). The examples’ timings are in examples/structure_edits.md and examples/structure_clusters.md.

Per-outcome coefficients and noise (2026-11-04)

A ClusterForm may give an entry of as an array and as an array: the factor’s coefficients and noise then depend on its outcome , as in Lenticulum’s RegimeFactor (src/regime.md). Given the assignment of the cluster’s discrete variables the continuous ones are still jointly Gaussian, but the precision

now depends on through the outcomes of the varying factors only ( holds the incoming messages and the factors whose and do not vary). The assignments are grouped into precision classes, one per combination of those outcomes; each class is factorised once (_precision_classes, the factorisation cache is keyed by class). Without a varying factor there is one class and every computation is the one above.

  • Messages. To a discrete channel nothing changes but the class-wise log-integrals. To a continuous channel each class gives its own precision , so the message is a mixture whose components need not share a precision. Components are merged when both their precision and their natural mean are equal. A proper component is normalised and its mass moved into the weight; an improper one is the unnormalised exponential its weight multiplies, which is how Mycelium’s mixture products read it (gaussian.jl, _improper_overlap). So improper components with different precisions are weighted consistently. (The first version refused them, after switching.md §5; that concern is about a factor that projects flat inputs out, not about an exact message.)
  • Belief and free energy. FrozenClusterBelief holds one covariance and one entropy per class and the class of each assignment. In the free energy a factor whose and vary is evaluated per outcome group: its residual means with , its trace term with the belief’s mass on the assignments of outcome in class , and its weighted by the mass on outcome . The negentropy is .
  • Plates (batched since 2026-11-12, Feature Backlog I6). The precisions depend on the parameters and on the inputs’ shape only, so tied replicas share them, the precision classes and their factorisations, as with one precision. What differs is the data term: a point shifts outcome ‘s residual by , so gets on the assignments of outcome , one column per assignment instead of one vector, and gets there (_batch_joint, per outcome group of _joint_plan). FrozenClusterBeliefs holds and per class and the class of each assignment; belief_free_energies evaluates each varying factor per outcome group over all replicas at once (its columns of in every replica, the trace term per class weighted by ), and EM’s _moment_stats adds with the mass of class . A factor that is not in a cluster but is a one-factor cluster (Lenticulum’s RegimeFactor) is batched the same way through as_cluster, its plate rows included in moment_statistics. What does not batch is what did not before: replicas whose inputs differ in shape (a different discrete observation, a different input precision, a categorical input with other labels: LP-032, where such a replica after the first was read positionally before the labels were part of the shape).
  • One factor outside a cluster. as_cluster(f) is a factor as a one-factor cluster over its own channels. A factor stated only by a cluster_form gets its messages and Bethe free energy from it (Lenticulum’s RegimeFactor does).

Tests: Lenticulum test/regime.jl. A mixture of factor analysers (, , three regimes with different and ) in one cluster gives to , the regime posterior and the three posteriors of (each with its own covariance) to ; the gradient in , , and the noise’s Cholesky factors agrees with central differences of to (ForwardDiff and Zygote); over a plate of four points the batched path is taken and is exact.

Over a plate (Lenticulum test/regime_plate.jl): on random regime factors (per-outcome on both channels, learned noise; six seeds, dimensions 1–3, 2–3 regimes, five replicas) the batched messages to every channel, under six input patterns (points, Gaussian inputs of one precision, a categorical input), equal the per-replica messages to , and the batched beliefs, graded free energies and their Zygote and ForwardDiff gradients equal the per-replica ones to ; a clustered mixture of factor analysers with random parameters on seven points, plated, gives to and the gradient of the closed form to (ForwardDiff and Zygote) from stacked beliefs; a mixture of regressions with one RegimeFactor per point and no cluster, plated and not, gives the closed-form to , equal gradients, EM statistics from the stacked beliefs equal to the per-node ones to , and the same EM path. Changing the per-assignment data term back to one vector makes 60 of the first testset’s 156 tests fail.

Flat directions (2026-11-04)

A direction of the continuous variables that no factor in the cluster and no incoming message determines is flat: the precision (, or when the target’s message eliminates the other variables) is singular there. Until 2026-11-04 the node refused it. It now integrates such directions out in the diffuse limit: a vague prior , , with the factor of the flat directions dropped. On the range of (an orthonormal basis , eigenvalues ),

which needs : every quadratic term vanishes on ( is a sum of positive semidefinite terms), so only a linear term of an incoming message can reach a flat direction, and that integral diverges; it is refused. The dropped factor is the same for all assignments only if they leave equally many directions flat; otherwise the node refuses (in the vague limit the assignments with more flat directions would take all the weight, which is rarely what a model means). This is the diffuse treatment of state-space models (de Jong 1991, the diffuse likelihood) and what the constraint factors do with a flat input (constraint.md §4.1).

  • Where it matters. A boundary channel with no incoming message yet (the first sweep of a loopy schedule), a factor outside a cluster with a flat input (as_cluster, Lenticulum’s RegimeFactor), and combinations of variables the model never determines (only enters, so is flat).
  • Belief and free energy. The frozen belief holds and per class, the flat directions at mean 0 and dropped; is the entropy on the range. Every factor’s annihilates , so the fit terms are the limits of the vague ones, and their gradient in at the frozen belief is the vague gradient’s limit (a term with vanishes).
  • Marginals. cluster_marginal of a variable in a flat class is the information form of the other variables eliminated; a variable with a flat direction gets an improper component (zero precision there) rather than a covariance with zeros.
  • Deciding flatness. An eigenvalue at most times the largest is flat. Cholesky can succeed on a singular (rounding leaves a pivot of , and the first test then gave variances of and a free energy 18.7 off), so its smallest pivot is checked against the same threshold. A precision that weak but meant (a prior of standard deviation beside factors of precision 1) is taken as flat, which shifts the free energy by the dropped constant.

Tests: Lenticulum test/cluster_flat.jl. With only determined, the free energy equals the vague graph’s () less to and the closed form to ; the switch posterior and the gradient (ForwardDiff, Zygote) equal the vague ones, and ‘s marginal is improper of zero precision. A RegimeFactor with seen through maps sends the message to the regime of the vague input to and the closed form to ; a linear term along a flat direction and regimes with different numbers of flat directions are refused; improper components of rank one with different precisions have the closed-form , and weights.

Frozen parameter paths (2026-11-04)

Until 2026-11-04 cluster refused clustered nodes with frozen parameter paths, so the structure examples held the sum node’s coefficients fixed by hand in their fitting loops. A clustered node’s freeze now carries over to the cluster node, under its parameter key: (key, path...) for each path, (key,) for a node frozen whole (or a factor that is isfrozen). The cluster node’s parameters are keyed by those keys, so the paths name the same coordinates, and the usual pinning (_factor_parameters) zeroes their gradient on every path, the batched plate path included. Tied groups must freeze the same paths. Test: Lenticulum test/cluster.jl, “Cluster nodes keep their factors’ frozen parameter paths” (zero gradient in the frozen A, the other gradients unchanged, ForwardDiff and Zygote).

Batching in one stacked joint (2026-11-13, Feature Backlog I19)

After I6 the batched path still built every replica’s joint as its own _ClusterJoint (a Dict of points, its data terms from loops over untyped plan entries) and its message by its own call of _joint_message; only the factorisations were shared. On the mixture of two regressions as a plate (bench/regime_plate_profile.jl, , before the change) a profile put about 45% of fit_em’s samples in building those joints, about 25% in the per-replica messages, and about 15% in as_cluster and _same_cluster over every replica on every sweep.

Now the replicas’ joints are one stacked joint (_batch_joint): replica 1’s structure (free variables, assignments, the class precisions ), and () and the constants () replica-major. Replica ‘s columns are replica 1’s plus the difference of their data terms, which are computed for all replicas at once: the points of a channel are the columns of one matrix , so per factor and outcome group , one solve , and and as matrix products (the formulas of “Per-outcome coefficients and noise” above, one column per replica). _joint_message is split into the parts that work on all columns of a joint (_continuous_parts: the target’s natural means and the log-weights, one set of solves per class; _discrete_parts: the log-weights) and the assembly of one message from one replica’s columns (_continuous_message, _discrete_message); _joint_belief likewise into _belief_parts, which normalises per replica. The per-replica path calls the same functions with all columns, so both compute the same thing. A plate row of factors that are equal fieldwise gets one shared cluster (_as_clusters), which _same_cluster accepts at once, and _same_struct compares fields through a generated, unrolled function.

Measured with bench/regime_plate_profile.jl (one machine, -t auto; logs in ~/.cache/lenticulum/i19/), before (at 1770cc13) and after, the same 11 iterations and the same to every printed digit:

fit_em beforeafterone iteration beforeafterallocated per iteration beforeafter
3000.338 s0.059 s52.8 ms10.5 ms64.9 MiB15.7 MiB
30003.611 s0.625 s621.6 ms140.9 ms647.1 MiB153.8 MiB

In Benchmark against RxInfer the plate then took 68 ms and 0.68 s against RxInfer’s 126 ms and 3.4 s. After the change the profile () puts 24% of the samples in the regime factors’ batched messages, 27% in the tasks the plate schedule steps one at a time (the data factors’ among them), 18% in the free energy; per replica there is still one message object to build and the grouping of equal components (_continuous_message). Both addressed by I20, below.

Tests: unchanged oracles (Lenticulum test/regime_plate.jl, test/cluster.jl: batched against per replica on random instances); mutating the batched data term (the quadratic term, the Gaussian inputs’ , the shift ) fails 90, 54 and 60 of their tests. From the review (LP-034 Findings of the review of I19): a row with an unequal factor goes replica by replica (regime_plate.jl), and flat directions in tied clusters over a plate agree with per replica and the closed form (cluster_flat.jl).

Block inputs (2026-11-15, Feature Backlog I20)

After I19 a plate sweep still went through the column path: the sweep holds the inputs as blocks (blocks.jl), the default factor_messages for blocks split them into one NamedTuple per replica, _cluster_batchable compared every replica’s input shape, _batch_joint stacked the columns again, and the result was one message object per replica, stacked once more by belief_block. Timed per group call on the mixture plate (, ~/.cache/lenticulum/i20/pieces.jl): splitting 1.1–1.5 ms, the shape check 0.2–0.4 ms, the stacked joint 0.4–0.7 ms, the per-replica messages 0.9–2.4 ms, re-stacking up to 0.9 ms; the solves of _continuous_parts under 0.1 ms.

Now factor_messages and factor_beliefs of a vector of ClusterFactors take a NamedTuple of blocks directly (RegimeFactor forwards to them). _block_batchable reads the shape off the blocks: points (on a discrete channel the same state in every column), Gaussians of one precision (shared, or per replica and all equal), categorical inputs to a discrete channel equal in every column, trivial or absent inputs. Anything else (mixture and vector blocks, inputs that differ between replicas) goes through the columns as before. _batch_joint reads the points of a channel as the block’s matrix and adds a Gaussian block’s to all replicas in one broadcast (_batch_points, _batch_eta!). The result is built as a block: a CategoricalBlock for a discrete target (_discrete_block); for a continuous target a MixtureBlock (a GaussianBlock for one assignment) when every class has one precision, no assignment has weight zero and no replica has two equal columns of (_continuous_block); otherwise each replica’s _continuous_message, stacked. That function now groups equal components by a linear scan over the replica’s few assignments with the precision classes deduplicated once (_precision_ids) instead of a Dict keyed by columns; the grouping rule is the same (isequal of the columns, equal precisions counted as one, order of first appearance).

The block path is not an approximation of the column path: the same arithmetic in the same order, so its blocks are equal bit for bit (Lenticulum test/regime_plate.jl, “block inputs give the column path’s messages exactly”: every block type it can return, merging, impossible states, several classes, differing inputs; four mutations of the new code fail 18 to 69 of its tests).

Measured with bench/regime_plate_profile.jl (~/.cache/lenticulum/i20/bench_base.txt and bench_b.txt, both 2026-11-15, the same 11 iterations and the same to every printed digit), together with the grouped data factors of plates.md “Untied factors without parameters” and a fit_em that no longer calls haskey on the parameter tree once per node:

fit_em beforeafterone iteration beforeafterallocated per iteration beforeafter
3000.057 s0.025 s10.3 ms3.9 ms15.7 MiB6.8 MiB
30000.615 s0.219 s138.5 ms27.7 ms153.8 MiB63.8 MiB

In Benchmark against RxInfer (2026-11-15, after I20) the plate took 30 ms and 0.24 s against RxInfer’s 126 ms and 3.2 s, with the same estimates as closed-form EM.

The profile after (): 51% of fit_em’s samples in the sweeps, of which most in the regime factors’ messages; 17% in the per-replica fallback of _continuous_block, which is the usual case once the M-step has given the outcomes different noises (the message to a continuous channel then has one precision per outcome, which a MixtureBlock cannot hold); 14% in _energy_rows, rebuilt by every free-energy and EM call. Both done in I21 (2026-11-15): with several precisions _continuous_block returns a ComponentMixtureBlock (blocks.md, “Mixtures with a precision per component”), component holding its class’s precision, and the energy rows are kept in the graph (free_energy.md). fit_em then takes 0.018 s and 0.132 s at and 3000 (~/.cache/lenticulum/i20/bench_c.txt), the same iterates; in Benchmark against RxInfer 19 ms and 0.15 s against RxInfer’s 125 ms and 3.4 s.

Derivatives at fixed messages (2026-11-16, Feature Backlog P24)

At fixed incoming messages the cluster’s free energy is

so its derivative in the parameters needs , not only the fixed-belief gradient that free_energy_gradient takes (LP-020). The two agree when the messages carry no information about the cluster’s free variables (only points, as in the clustered mixture of factor analysers: is then constant under and the first term drops out), and differ otherwise (the P24 testset of Lenticulum test/regime_plate.jl checks that they differ, by more than , with Gaussian messages).

  • ForwardDiff. The joint (_cluster_joint, _enumerate_assignments, _log_integrals, _belief_parts, _continuous_parts) is generic in its number type: _ClusterJoint{T} with T the promotion of Float64, the forms’ and the incoming messages’ number types (_joint_numtype). The Cholesky factorisations are LinearAlgebra’s generic ones. The stacked joint of a plate (cluster_batch.jl) stays in Float64: parameters of another number type (_plain(ps) false) go replica by replica.
  • Zygote. The joint is built by enumeration with push! and in-place writes. Instead of rewriting it without mutation, _cluster_local_energy (which local_free_energy of a ClusterFactor and of a RegimeFactor call) has an rrule in ext/MyceliumForwardDiffExt.jl (loaded with ChainRulesCore and ForwardDiff; Zygote loads both) whose pullback takes ForwardDiff.gradient of the graded parts, weighted by the cotangent, over the flattened parameter tree. A cluster node has few parameters, so the forward passes are cheap; on a plate, P20’s row rule already differentiates replica by replica, and each replica runs this rule. The incoming messages get a @not_implemented tangent: inside AD they would come from cluster messages, which Zygote cannot take either.
  • Found on the way. On a plate row Zygote then stopped at the state write-back: _writeback records the empty state of a key that had none with a push!, whose reverse pass Zygote refuses. A row’s state is empty or tied and stateless, so no tangent flows through it: _group_writeback is @non_differentiable (ChainRulesCore extension); the primal still writes it (the state tree keeps the key, which callers read).
  • Tests (Lenticulum test/regime_plate.jl): five random regime factors with Gaussian messages on both continuous channels and a categorical one on the switch, alone and clustered with a prior (energy at the messages of one sweep): ForwardDiff against central differences of the value path (1e-6), Zygote against ForwardDiff (1e-10), on the flattened vector and on the NamedTuple with every graded part weighted; and the clustered mixture of factor analysers through scalar_free_energy on the store with block rows (were @test_broken).

Flat directions under a derivative (2026-11-16, Feature Backlog P26)

The diffuse limit (“Flat directions”, above) splits by an eigendecomposition, which has no rule for dual numbers; until P26 ForwardDiff (and so Zygote, whose rule runs ForwardDiff, “Derivatives at fixed messages”) refused a cluster whose joint precision has flat directions. The quantities the limit uses, , and the range, are smooth on the positive semidefinite matrices of one rank , whose tangents at are the symmetric with ( a basis of ). With on the range, the range and the null space move only into each other to first order,

both orthonormal to first order (check: and ), and , since .

  • Implementation. _Flat holds , factorised (a Diagonal of the eigenvalues for floats, a Cholesky for duals) and ; _psd_solve, _psd_inv and _psd_logdet read . _psd_factor hands a that Cholesky does not take to _flat_factor: for floats the eigendecomposition as before; for first-order ForwardDiff duals of a float the extension MyceliumForwardDiffFlatExt (loaded with ForwardDiff alone) decides flatness on the values, then builds , and from the duals, as above. No eigenvector is differentiated, so eigenvalues that repeat on the range are no difficulty.
  • Rank changes are refused. A partial with makes a flat direction determined to first order; there the limit drops a different constant and has no derivative, and the extension throws an ArgumentError. A direction that becomes determined only at second order passes the test: at (a variable that enters no factor) the cross block of in is nonzero but . The value is discontinuous there: once leaves 0 by more than the flatness threshold lets pass, the direction counts as determined and the free energy goes like (a scratch run of P24’s example, 2026-11-16, not kept as a test). The derivative given in is the one along the matrices of constant rank, not of the value path. A model whose flat directions are flat for every parameter (a map with fewer rows than columns, below) is not affected.
  • Other number types. Nested duals (a Hessian) and other non-float types still throw an ArgumentError naming P26 on the flat path.
  • Tests (Lenticulum test/regime_plate.jl, testset “flat directions under a derivative”): twenty random of rank with a rotating range, on half of them (all eigenvalues on the range equal), _psd_logdet, _psd_solve and _psd_inv through ForwardDiff against central differences ( relative); a rank-changing tangent refused; five random regime factors whose is wider than their output and has no message ( flat directions in every regime, checked), ForwardDiff of the free energy against central differences () and Zygote against ForwardDiff (); and P24’s former @test_broken () in the parameters other than .

What it does not do

  • Factors outside the conditionally linear-Gaussian class. A factor needs cluster_form (its and may depend on its outcome since 2026-11-04, above). DataFactor stays outside: data reaches the cluster as point messages.
  • Flat directions with unequal counts, or reached by a linear term. Refused (above). Flat directions in equal number are integrated out in the diffuse limit.
  • Enumeration is exponential. At most joint discrete states of nonzero weight (pruned as they are built, above); equal components of a message are merged, but nothing reduces the number of distinct ones. Precision classes add one factorisation each.
  • Plates, other rules, other couplings. cluster(g, groups) refuses graphs with plates (cluster(g, plate, name, factors) clusters a plate’s factors, on a graph with one plate), nodes that are not on belief propagation, a parameter key tied to a node outside, and edges that are not bidirectional, diagonal and attached. The cluster node itself is a belief-propagation node; differentiating through inference (adjoint = :implicit) is not tested with it.
  • AD where the rank of the joint changes. ForwardDiff (and so Zygote) through flat directions is the derivative at constant rank (above); a tangent that changes the rank to first order is refused, one that changes it at second order ( leaving 0) gets the constant-rank derivative, and nested duals are refused. No tangent reaches the incoming messages under Zygote (“Derivatives at fixed messages”).
  • Grouping rewires nothing. overlap_groups chooses the groups; putting a partial sum per group into the graph, which is what makes the groups pay (below), is an edit of the graph. For switches feeding a Lenticulum linear factor, Lenticulum’s partial_sums makes it (src/switching.md §14); Mycelium itself knows no sum factor.

Difficulties

  1. Tied parameters. The first version keyed the node’s parameters by factor name and refused tied factors, which refused the switching chain at once (its observation and dynamics factors are tied). The node now keys them by parameter key, and refuses only a key shared with a node outside the cluster.
  2. A comparison the test could not make. A cluster of the three switches and the block factor were first compared by the graph free energy; neither graph has one, since the have mixture marginals and Mycelium no mixture entropy. The test compares the nodes’ local free energies instead, and a separate test shows that clustering the in removes the gap.
  3. Review (Gemini via agy, 2026-11-03). Three findings were fixed: a factor channel left unconnected inside a cluster gave a KeyError (now refused by cluster, tested); a discrete input with no possible state gave an empty-collection error instead of an ArgumentError; the batched mixture product did not check the replica count. Two were not bugs: _indefinite depends on the precision only, so a shared one is one test; and Zygote through belief_free_energy works (the chain’s gradient is tested with it, and _cluster_layout is non-differentiable in the ChainRulesCore extension, which the reviewed diff did not show).
  4. Review of the grouping (Gemini via agy, 2026-11-03). Fixed: a cluster’s parameter key could equal the key of a node left outside, and cluster_parameters would then overwrite that node’s entry (refused now, tested); tied nodes compared their parameter keys in order, so the same factors listed in another order were refused (compared as sets now); switch_overlaps took the directions of a switch with three or more outcomes from the first outcome only (every pair now). Documented: the metric counts the noise of the sum and of the listed switches only.

Review of 2026-11-05 (Mistral via vibe)

A read-only review of the EM code found that batched cluster nodes treated an observed discrete boundary variable as “a point” without its state (_input_shape). Replicas that observed different states were batched on replica 1’s surviving assignments, so their messages, free energies and EM statistics were those of replica 1’s state. Reproduced in Lenticulum test/cluster.jl (“Batched cluster nodes with an observed switch that differs between replicas”): a batched free energy of 54.76 against the exact 71.59, and the statistics’ masses [6, 0] against [3, 3]. The observed state is now part of the shape, so such replicas go replica by replica. The defect dates from the batched path (2026-11-04); the examples observe no discrete variable, so their results are unaffected.