Mycelium
Factor graphs and message passing — how factors are wired and in what order they talk. This is the layer with no counterpart in Lux, because in Lux the wiring is a DAG and the order is implied by it.
The docstrings of every exported name are on the Mycelium API page.
Building a graph
b = GraphBuilder()
variable!(b, :x, 1)
variable!(b, :y, 1)
factor!(b, :f, some_factor)
connect!(b, :f, :in, :x; direction = Bidirectional())
connect!(b, :f, :out, :y; direction = Bidirectional())
g = validate(build(b))validate checks the structure: every edge names a channel its factor actually declares, nothing is isolated, and dangling outputs are reported. It is a separate, optional step.
connect!(...; coupling, detach=false) accepts all four built-in couplings. factor!(...; freeze=(:encoder,)) pins a subtree, several tuple paths pin several, and freeze=:all pins the full tree. Frozen coordinates have zero graph gradient and no optimiser state; input derivatives are preserved. isfrozen=true pins all parameters for this run; islearnable=false excludes an intrinsically nontrainable factor from learnable_factors. Frozen learnable factors remain in that candidate list.
Edges are directed, and the direction restricts which messages are legal:
| direction | the factor can |
|---|---|
Bidirectional() | both send and receive |
Emitting() | only send (a data clamp) |
Absorbing() | only receive (a loss) |
Two acyclicity notions
Do not confuse them:
| predicate | about | decides |
|---|---|---|
istree(g) | the undirected graph | whether message passing is exact |
isdag(g) | the directed multigraph | whether the graph is Lux-compatible |
They are independent.
Running inference
marg, report, st, store = infer!(g, tree_schedule(g), ps, st)marg is a NamedTuple of marginal beliefs keyed by variable name; report says whether messages converged; store holds the messages, and is what the free-energy functions read.
Schedules: tree_schedule (exact on trees), flooding_schedule (general, iterative), forward_backward_schedule (chains — the Kalman/RTS smoother), SequentialSchedule.
A factor node computes its messages by belief propagation unless it is added with factor!(b, name, f; rule = MeanField()). Such a node sends $\exp \mathbb E_q[\log f]$ over the other variables' current marginals (mean_field_message, implemented by the factor). Mixed with belief-propagation nodes this is the hybrid BP/mean-field scheme of Riegler et al. (2013). hybrid_schedule builds its iteration from the rules (each belief-propagation cluster swept around the mean-field nodes, which are updated in turn), and propagate! runs it to a fixed point measured on the marginals; a hand-written SequentialSchedule works too. The graph free energy is then the merged free energy of that paper, eq. (17): a mean-field node contributes mean_field_energy, its average energy under the product of the marginals, and the variable counting numbers count belief-propagation neighbours only.
Two more rules handle factors whose messages leave the Gaussian family, such as switching factors, whose messages are mixtures. ExpectationPropagationRule sends the belief-propagation message projected the EP way: the target's cavity times the message, moment-matched to a Gaussian, divided by the cavity (ep_message); updates without a proper cavity are skipped, and the graph is iterated to a fixed point, where its free energy (counted like belief propagation's) is stationary. MixtureReduction(M) keeps mixtures: mixture inputs are expanded exactly (mixture_message, implemented by Lenticulum's linear and switching constraint factors) and the message is reduced to M components by Runnalls' merge (reduce_message). On a switching chain the budget moves the answer from assumed-density filtering to the exact posterior; such nodes have no free energy.
cluster(g, name, factors) replaces factor nodes by one node with exact inference inside: its discrete states are enumerated and its continuous variables eliminated, for factors that state themselves as a residual with Gaussian noise and outcome weights (cluster_form: the constraint factors and Gaussian or categorical priors). Outside it is one factor with exact messages and a closed-form local free energy; cluster_parameters rearranges the parameters and cluster_marginal reads its internal variables. A whole model in one cluster is exact (its free energy is $-\log p$ of the data), and clustering a mixture variable with its switches gives it the free energy plain belief propagation lacks. Several clusters are made in one call, tied with params (one per data point of one model); overlap_groups groups switches by Lenticulum's switch_overlaps, as E7 did, and Lenticulum's partial_sums adds E7's partial sum per group as an edit. TableFactor is a fixed factor over several discrete variables; its -Inf entries tie them together (a switch's outcome to a cluster label), and a cluster node drops the assignments they rule out before its Gaussian part. On a plate, cluster(g, plate, name, factors) makes one tied node per replica and keeps the plate, and tied cluster nodes then share the work of their messages and free energies; as_plate gives an edited graph its plate back. A factor's coefficients and noise may depend on its outcome (one precision per class of assignments, as Lenticulum's RegimeFactor), and as_cluster makes one factor a cluster node of its own.
A built graph is edited by opening it as a builder: GraphBuilder(g), then variable!, factor!, connect!, remove_factor!, remove_variable!, disconnect! and replace_factor! by name, then build. transfer carries the messages of the edges both graphs share into a store for the new graph, as a warm start. lib/Lenticulum.jl/examples/structure_edits.jl runs a greedy structure search over switch wirings this way, scored by the exact free energy.
Points can be passed instead of beliefs. GibbsSampler(g, sampled) draws the named variables (for example the discrete switches) from their exact conditionals, integrating the others out by belief propagation with the drawn ones pinned (pin!). A SampleBuffer keeps the samples, weighs them by a WindowFilter or DecayFilter, forgets those older than a number of parameter steps (advance!), and reports per-component energy moments, each factor's residual mean and full covariance, and the Monte Carlo EM gradient (sample_gradient: one inference per sample, then the fixed-belief gradient).
Structural factors
Data, priors, losses and optimisers are all factors here rather than special machinery:
DataFactor, PriorFactor, LossFactor, RelayFactor, OptimiserFactor, and the update rules GradientDescent, Momentum, Nesterov.
Free energy
scalar_free_energy, bethe_free_energy, factor_free_energies and variable_corrections compute the graph's objective — a sum of per-factor energies plus a per-variable entropy correction. On a converged linear-Gaussian tree this equals the exact negative log marginal likelihood; loopy graphs use the Bethe approximation; see the worked example.
uses_cavity(factor) defaults to false; implement it as true whenever local_free_energy reads incoming variable-to-factor messages. All graph free-energy entry points reject Emitting() edges on those factors with ArgumentError, identifying the factor, channel and variable and recommending Bidirectional() (LP-014). Missing messages on Absorbing() or Bidirectional() edges keep their existing behaviour. PriorFactor and LossFactor return true; the constant-zero local free energies of DataFactor, RelayFactor and OptimiserFactor retain false.
combine_with_overlap(c,d) returns the pooled belief and log overlap for categorical and Gaussian pairs. A Gaussian first argument must be proper; an improper second argument is allowed up to a constant depending on it alone. Disjoint categorical support returns (c,-Inf). Mixture weights and the AlgebraicFactors homotopy extension use this public rule.
Parameter steps
After inference has populated incoming messages:
using DifferentiationInterface, ADTypes, ForwardDiff
ad = AutoForwardDiff()
grad = LenticulumCore.free_energy_gradient(store, g, ps, st; ad)
ps, st, optstate = parameter_step(store, g, ps, st; ad, rule=GradientDescent(0.01))
# On subsequent steps pass optimiser_state=optstate (needed for Momentum/Nesterov).The objective is the sum of scalarised local free energies, with beliefs fixed. Bethe variable corrections have zero parameter derivative under that convention. Only learnable factors are updated. Inference schedules and OptimiserFactor remain separate; no gradient is propagated through the mutating inference schedule. LuxFactor requires residual=true to learn from endpoint scoring in this mode. The ImplicitLayers AD tests fit a known ODE coefficient and check Nesterov look-ahead updates with ForwardDiff and Zygote. The diagonal parameter driver also has closed-form state-dependent SciML endpoint tests with ForwardDiff, Zygote, Mooncake and runtime-activity Enzyme reverse. NeuralODE factors opt into state_policy=:fixed to hold incoming Lux state constant per solve and obtain returned state from one final endpoint evaluation. The factor validates its solver/backend combination before differentiation; see the ImplicitLayers package for tested solvers and the guarded Kvaerno3/InterpolatingAdjoint limit. This extends endpoint coverage without changing schedule derivatives.
Graph schedule adjoints (LP-002 batch C)
For Exact/Pathwise edges, finite replay now supports sequential, tree, forward/backward, flooding and residual schedules, including damping:
grad = LenticulumCore.free_energy_gradient(store, g, ps, st;
ad=AutoForwardDiff(), schedule=flooding_schedule(g), nsweeps=20, damping=0.3)Replay starts from empty messages. Flooding reads one snapshot per sweep. Residual replay records the chosen task sequence in a reference pass, then differentiates that fixed sequence. This is a derivative of the realised piecewise schedule; argmax, ties and convergence decisions have no derivative. The raw Zygote infer! rule also replays its actual count/choices and damping. Score estimates retain the one-undamped-static-sweep restriction.
At a converged BP root, opt into a message-space implicit adjoint:
using Zygote
grad = LenticulumCore.free_energy_gradient(store, g, ps, st;
ad=AutoZygote(), schedule=flooding_schedule(g), adjoint=:implicit,
maxsweeps=1000, tol=1e-10, damping=0.0,
linear_solver=:dense, linear_tol=1e-10, linear_maxiter=1000)
# using Krylov; linear_solver=:krylov selects matrix-free GMRES.
# parameter_step forwards the same keywords.The reference schedule finds a root from empty messages; nsweeps controls only the unrolled driver. The implicit map is synchronous BP with outgoing damping. It solves (I-DT)'λ = ∂L/∂m and returns ∂L/∂ps + (Dps T)'λ, including Bethe corrections in L. This requires a differentiable local root and invertible I-DT; a contraction suffices. Convergence is checked against every BP row and every linear solve checks its original residual. These checks do not prove uniqueness or conditioning.
Diagonal/detach edges hold their legal directed update rows constant: their message variations are zero in the linear system. Frozen parameter coordinates are pinned while message derivatives through the factor survive. With mixed cuts, this agrees with the limit of cut replay when its derivatives converge. Exact/Pathwise only, with CPU real numeric Dirac or Gaussian messages (dense or diagonal precision) and recursively empty message state. Reference differences, ForwardDiff and Zygote are tested. Native Enzyme/Mooncake, implicit score sampling, device coordinates, evolving message state and higher-order graph adjoints are unsupported.
Dense solves store a quadratic message Jacobian and have cubic factorisation cost. GMRES avoids that matrix, retains at most 30 Krylov vectors and uses a backend gradient per transposed product. With ForwardDiff that gradient still scales with message dimension. Both methods also pay reference inference and functional store-copy costs; neither differentiates the reference iterations. Finite replay retains its sweep-dependent tape/copy cost.
lib/MyceliumLearning.jl/test/graph_adjoint.jl supplies dense Gaussian tree oracles, reconverged loopy Bethe differences and unrolled convergence measurements. graph_adjoint_devices.jl tests the new replay with diagonal Gaussian JLArrays in Float32/Float64 and scalar indexing disabled; implicit device calls reject both linear solvers because the coordinate adapter is CPU-only. No hardware GPU result is claimed. LP-002 and the implementation notes record measurements and the native-backend and metric/divergence reproducers.
Plates: many observations, shared parameters
A plate replicates a template over observations in one graph. Factors inside it are tied: every replica reads one entry of ps, and its gradient is the sum over replicas. A name the template does not declare is a node outside the plate, shared by every replica.
b = GraphBuilder()
plate!(b, :obs, N) do p, i
variable!(p, :z, k); variable!(p, :x, d)
factor!(p, :prior, GaussianPrior(:z, zeros(k), Matrix(1.0I, k, k)))
factor!(p, :lik, LinearConstraintFactor((x = d, z = k), d; noise = 0.7)) # ps.lik, shared
factor!(p, :data, DataFactor(:x, X[:, i]))
connect!(p, :prior, :z, :z); connect!(p, :lik, :z, :z)
connect!(p, :lik, :x, :x; direction = Absorbing()); connect!(p, :data, :x, :x; direction = Emitting())
end
g = build(b) # nodes :z[1], …, :z[N]; parameter_keys(g) == (:prior, :lik, :data)
sched = plate_schedule(g) # each template task for all replicas at once
_, st = propagate!(MessageStore(g), g, sched, ps, st)Any node can be tied outside a plate too, with factor!(b, name, f; params = key). Tied nodes need the same factor type and freeze. plate_schedule(g, replica_tasks) lifts a hand-written schedule for replica 1; a template with mean-field nodes gets its hybrid_schedule by default. A sweep of a plate schedule gives the messages of the same tasks run one by one. The store holds each template edge's messages as one BeliefBlock (Gaussian information vectors as a $d \times N$ matrix with a shared precision, points as a matrix); reading store.to_variable[e] still returns that edge's message. A factor computes a group's messages on blocks through factor_messages, and a plate's free energy and gradient through local_free_energies, factor_beliefs and belief_free_energies; LinearConstraintFactor and GaussianPrior implement them, and SwitchingConstraintFactor too, with its messages to the outcome as one CategoricalBlock (a Gaussian-mixture E-step over 1000 points: about 2.3 times the closed form) and its mixture messages as one MixtureBlock; a plate of expectation-propagation nodes sends its sites as one block (ep_block).
Known gaps
combineis partial: particles can be reweighted by a density or anAbstractPotentialBelief.LogPotentialBeliefwraps a callable log-density; two potentials combine by adding log-densities. Two particle clouds still cannot be pooled directly. Gaussian, categorical, and mixture products with an arbitrary potential throw rather than resample; structured belief pairs retain their explicit rules.- Potential distances compare callable identity (otherwise
Inf); damping potentials is an inspectable no-op throughcan_damp. - Plates are batched for Gaussian, point, categorical and mixture messages (factor analysis: about 9 times the E-step written by hand at $N = 1000$; a Gaussian mixture's E-step about 2.3 times;
blocks.md), and for expectation-propagation nodes on blocks. Mixtures whose components differ in precision, other beliefs, andMixtureReductionnodes go one replica at a time. - Belief-propagation graphs whose marginals are Gaussian mixtures have no free energy (no closed-form mixture entropy) unless those variables are inside a cluster node;
MixtureReductionnodes have none either. Cluster nodes enumerate their discrete states (at most $2^{16}$) and take conditionally linear-Gaussian factors only. - In the coupled drivers (
adjoint = :implicit, non-diagonal couplings), a factor whose edges are all cut is held at its fixed belief, as in the default driver (LP-021); a factor with some edges cut and some not is differentiated at its messages, with the cut ones constant. free_energy_gradientandparameter_stephold each factor's belief fixed (the envelope gradient, exact at a belief-propagation fixed point) only for factors that implementfactor_belief: Lenticulum's Gaussian, constraint and switching factors, and AlgebraicFactors' toric and degree-one polynomial factors. Other factors (the neural and diffusion ones) keep the fixed-message derivative, which is exact only with point messages, and warn otherwise (LP-020).- Graphs with
MeanField()nodes have the merged free energy of Riegler et al. (2013), eq. (17) (L-SPEC-005), andparameter_steptrains them; differentiating through inference (non-diagonal couplings,adjoint = :implicit) and the sample buffer still refuse them. - Loopy message passing has no convergence guarantees, and
report.convergedmeans the messages stopped moving, not that the answer is right.
Game quantities and conformance
Data and Relay have zero game free energy and entropy. Prior supports exact Dirac and categorical games; arbitrary Gaussian nlogp expectations still throw. Loss is a terminal game: pass every channel as a point or Dirac in the named observation y; its entropy is zero with the loss's shape. Optimiser is an update rule and both game methods throw ArgumentError. Structural factors use the shared checker in test/conformance.jl: direct messages for emitting factors (both Relay directions), explicit pointwise/game opt-outs for Optimiser and sink/message/pointwise opt-outs for Loss. Their supported game methods are compared to local_free_energy using option 1.
Finite schedule gradients and straight-through values
connect!(builder, :model, :out, :y; coupling=ExactCoupling(), detach=false)
factor!(builder, :encoder, encoder; freeze=(:pretrained,))
grad = free_energy_gradient(store, graph, ps, st;
ad=AutoForwardDiff(), schedule=tree_schedule(graph), nsweeps=1)The diagonal default holds each factor's belief fixed (LP-020): for factors with factor_belief, which include Lenticulum's Gaussian ones, it is the fixed-posterior EM derivative, and at a fixed point the gradient of the Bethe free energy. Other choices replay from empty messages, with explicit finite sweep count. A derivative path survives only if every crossed attachment permits it: exact/pathwise keep their local Jacobian, diagonal cuts it, and detach overrides all. Pathwise supports Dirac and analytic Gaussian moment propagation; other informative families throw. Exact differentiates the iteration, including Bethe corrections, without promising converged/exact inference in loopy graphs.
Static sequential/tree/forward-backward schedules and DI ForwardDiff/Zygote or reference differences are supported. Reverse infer! replays its realised static sweep count with zero damping. State/reports have no cotangents. Flooding/residual replay, other schedule backends and GPU are unsupported. Replay storage/copying grows with tasks times edges.
ScoreFunctionCoupling(; baseline=nothing) supports sampled terminal Gaussian/categorical messages on all-absorbing factors for one sweep. Supply an explicit rng and nsamples >= 2; GradientEstimate contains .value, componentwise .standard_error, and .nsamples. Its objective is expected terminal point cost plus the other analytic local/Bethe terms; sampling a terminal Dirac does not estimate distribution entropy. Categorical probabilities must be strictly positive. The baseline is a fixed finite scalar. Score parameter_step returns four entries, including the estimate; deterministic steps retain (ps_new, st_new, optimiser_state). Coupled steps commit the first replicate's inferred state plus one local-energy advance; other replicates do not accumulate state changes. Bidirectional score sampling throws because replacing BP densities by samples changes the free-energy objective. Mixed terminal score and upstream analytic/cut rules compose.
StraightThroughFactor(round, identity) has (:in,:out) channels, forward round values, identity input Jacobian and zero local cost. Its fixed callables take scalar/array inputs. ForwardDiff and ChainRules reverse mode are supported; the graph derivative entry points reject other backends, including differences of the forward function.
The types and functions are listed under Gradient couplings in the API reference.
Device graph gradients
JLArrays tests cover Float32/Float64 static replay with ForwardDiff/Zygote for Dirac and diagonal Gaussian messages: exact/pathwise/diagonal couplings, detach, subset/all freeze, raw static inference AD and parameter_step. Active-coordinate helpers use device gathers and broadcast masks. GradientDescent/Momentum/Nesterov cast configuration coefficients to device parameter precision; CPU rules stay as before. Terminal diagonal-Gaussian score estimates and standard errors match CPU with common host RNG draws bulk-copied to the device. Dense Gaussian devices, categorical score devices, other schedule backends and hardware GPUs remain untested.
Implicit DEQ messages in explicit-backend replay use IFT solves, including reverse input messages with empty state. Both DEQ and Euler message directions have nonzero downstream-loss tests; raw Zygote replay also selects the implicit rule. For ForwardDiff implicit graphs, use the explicit free_energy_gradient driver rather than relying on raw infer! to select IFT. Reverse-input DEQ replay with nonempty Lux state is rejected because primal threading and fixed-state IFT differ.
Raw ForwardDiff DEQ messages now reject dual-valued unannotated solves with an ArgumentError pointing to the explicit graph gradient driver, preventing accidental root-iteration differentiation. Ordinary primal messages are unchanged.
The existing GaussianFactor, GaussianPrior and GaussianComposite replay adapters accept that backend context too, preserving their functional matrix-inverse formulas and closed-form CPU marginal-likelihood gradient checks.