implementation

DEQFactor: the fixed-point condition kept as a residual, so the polarity decides which channel to solve for.

Sources: original residual/adjoint derivations; code: deq.jl, ../test/solver_numerics.jl

Theory (CT-ML wiki): Bayesian Inversion · Bayesian Lens · Variational Free Energy · Lens

Graph free-energy edge directions

DEQFactor declares Mycelium.uses_cavity == true: its local free energy evaluates the residual at incoming points, even though it returns zero if a point is absent. Graph free-energy evaluation rejects an attached Emitting() edge with ArgumentError; use Bidirectional() if that channel must also send messages. Missing messages on permitted edges retain their existing local behaviour. See free_energy and LP-014 Emitting edges drop the cavity message from the free energy.

1. What SciML gives you, and what it keeps

DeepEquilibriumNetworks.jl builds a Lux layer. You hand it a cell and a NonlinearSolve.jl solver, and you get x ↦ z★(x) — the relation solved in one direction, with the solve sealed inside a function and the solve’s diagnostics tucked into st as a DeepEquilibriumSolution.

Everything in that sentence is good engineering and none of it is a relation any more. The object that is a relation is the residual:

which is README’s definition of an implicit learner, verbatim. DEQFactor keeps it, and therefore gets both directions out of one solver:

polaritysolveswho can do this
x observed, z unobserved for SciML’s DEQ, and this
z observed, x unobserved for only this

Same residual, same BroydenSolver, different variable held fixed. The test suite checks both against the closed form for a linear cell, and checks that they really are inverse to each other.

2. The energy is a proper vector energy

pointwise_energyspace is EuclideanEnergySpace(dim z) and pointwise_scalarisation is SquaredNorm — the residual is returned as an -valued object, not collapsed to a number.

This is what Scalar and Multivariate Energy asks every implicit factor to do and what VariationalDiffusion.jl’s factor could not deliver (its predictor.md §4.3 records the inconsistency). Here it comes for free, and it is exactly what the IFT needs — that note’s §6.3 is about needing the Jacobian of the vector residual.

The game energy instead has ScalarEnergySpace() and IdentityScalarisation(): it contains the squared residual, not the residual vector.

3. The reverse direction needs a square residual

Solving for x is equations in unknowns. Unless it is over- or under-determined, and a root-find is not asking a well-posed question.

supports_polarity refuses that polarity when the dimensions differ, and supported_polarities returns one element rather than two — so a non-square DEQ correctly reports isunidirectional == true, i.e. it really is just a Lux layer. Asserted in the tests.

The honest reading: the bidirectionality of a DEQ factor is conditional on a shape coincidence. A DEQ whose input and state have different widths — which is the common case in practice, since the input is usually an embedding — is unidirectional and gains nothing from this package over LuxFactor. What would help is a least-squares solve for the over-determined case and a regularised one for the under-determined case; neither is implemented.

4. Implementation difficulties

4.1 Every message is a DiracBelief

A root-find returns a point. Propagating a distribution would need the linearisation deq_sensitivity already computes — push a covariance through — and somewhere to put it, i.e. a GaussianBelief.

GaussianBelief used to live in the top-level Lenticulum package, out of reach of every lib/ package. It has moved to LenticulumCore (October 2026, The Equilibrium Family §5), and VariationalDiffusion already returns Laplace Gaussians. For this factor the same is possible, from deq_sensitivity, but not built.

4.2 The prior warm-starts the solve

The prior argument supplies the solver’s initial guess. The returned Dirac site does not condition on that prior; likelihood_message therefore uses the default point-lens path. A conflicting hard prior cannot be combined with that site. The lens game below records that incompatibility as infinite anchor cost; it does not change the solve into Bayesian conditioning. No distributional posterior is supplied.

4.3 Point-belief entropy convention

local_free_energy returns without an entropy term. The game adopts the same zero point-belief entropy convention as Mycelium’s counting correction; this is not the differential entropy of a Dirac relative to Lebesgue measure. The graph method can also read a Gaussian’s mean, but supplies no joint distributional factor entropy. Consequently, mixing these residual diagnostics with Gaussian factors does not establish a normalized marginal-likelihood interpretation for the resulting Bethe objective.

4.4 The initial guess is zeros

solve_state starts from unless warm-started. SkipDeepEquilibriumNetwork exists precisely because a learned initial guess converges much faster and regularises training. That is a second parameter tree on the inversion — the AmortisedInversion situation of Inversions and Bayesian Lenses — and implementing it would make this the first factor in the project with genuinely two parametrised halves. Not done.

4.5 Sensitivities and training

deq_sensitivity(f, x, zstar, ps, st; wrt=:x, ad=nothing) defaults to central differences; wrt=:ps selects flattened parameters. Supplying an ADTypes backend uses DI residual derivatives. solve_state(...; ad) also has optional ChainRulesCore, ForwardDiff, EnzymeCore and Mooncake extensions implementing the IFT through the solved state; see solve. DEQFactor(...; ift=DenseIFT()) keeps dense LU as the default. ift=KrylovIFT(; rtol, atol, itmax, memory, precond) selects matrix-free GMRES in all solved-state AD rules (load Krylov and DI); failure to converge throws. deq_sensitivity remains a dense reference regardless of this option. See solve for the operator products, storage, backend choices and 200-dimensional oracle. Its incoming state is fixed throughout iterations and advanced once at the root.

For , a perturbation obeys

test/parameter_ad.jl checks this parameter Jacobian against ForwardDiff, Zygote, Enzyme, Mooncake and central differences, and checks a scalar loss on the solved state. energy_gradient and free_energy_gradient instead score fixed incoming endpoints, so their DEQ parameter derivatives need no root solve. Graph parameter_step uses this latter objective with fixed messages; it does not backpropagate through message schedules.

Related: solve, luxfactor, neuralode, DEQ as a Relation, The Equilibrium Family, Scalar and Multivariate Energy, Implicit Learners

Incoming factor game and solved-point lens game (LP-009)

free_energy(f, π, y, ps, st) follows GaussianFactor’s fixed channel order: π is the incoming point on the first channel (x), y the point on the second (z). This corresponds to the reverse split, with x unobserved and z observed; the factor method takes no polarity and does not reinterpret the arguments for a forward solve. Both must be DiracBeliefs. The method passes them to local_free_energy; entropy is scalar zero. Other beliefs throw because no distributional residual expectation is defined.

lens = first(assemble(f, p, ps, st)) restricts the relation to polarity p. free_energy(lens, π, inputs, ps, st) runs that inversion and evaluates the residual at its returned point, plus the prior cost on the unobserved channel. Here inputs is the named Dirac observation (or that Dirac directly). A trivial prior costs zero, a Dirac costs zero only at the same point and infinity elsewhere, and a proper Gaussian costs its quadratic penalty relative to its mean. No normalizing constant is charged. This is where Definition 20’s expectation under the inversion lives, with zero point-belief entropy. A finite channel precision does not soften a Dirac prior; soft observations and singular Gaussian anchors throw. The solver still uses the prior only as a warm start.

For residuals affine in the unknown u, with root u*,

Thus the incoming factor energy equals the solved residual energy (zero at a root) plus an anchor of precision J_u'J_u. energy_jacobian supplies this metric; the caller chooses it, and the lens game does not invent a prior. For g(z,x)=z/2+x, at x=1,z₀=3, the root is z*=2, Jz=1/2, so both sides are ½(1/2)²=1/8. In reverse at z=3,x₀=1, the root is x*=3/2, Jx=-1, giving ½(1/2)²=1/8 again.

test/conformance.jl checks both full factor games without game_gap, solved-point lens games, hard-prior infinity, and the exact linear identity. Its nonlinear cell has r= z+z²/4-x, root z*=2 at x=3, and Jz=2. At z₀=2+h the incoming energy is 2h²+h³/2+h⁴/32; the Gaussian anchor costs 2h². The gap is h³/2+h⁴/32, checked at four halving step sizes. The residual approximation is first order (Gauss–Newton), with cubic absolute energy error and, generically, first-order relative energy error near the root.

Scoring an arbitrary point uses energy/scalar_energy, the pointwise level of L-ADR-0003 Pointwise energy and free energy are separate levels, not a keyword on free_energy. Shared implementation and limitations: point_game.

Solved-state IFT AD accepts AutoForwardDiff(), AutoZygote(), AutoEnzyme() and AutoMooncake(). Plain AutoEnzyme() works for these DEQ rules without runtime activity; the built-in ODE stage fold requires an explicit runtime-activity mode (see neuralode). Load DI, the backend and (for Enzyme/Mooncake) ForwardDiff; missing extensions and other solve backends are rejected before solving. Pass the backend to solve_state and the outer derivative. Enzyme outer closures that capture arrays need DI constant contexts or function_annotation=Enzyme.Const; see solve for the limitation and reproducer. Hardware GPU execution remains untested; the JLArrays subset is described below. Second order uses the dedicated dense deq_hvp API below.

Solver numerics and the other polarity (2026-10-27)

LimitedBroydenSolver(; memory=20, maxiters=100, tol=1e-10, init_scale=1.0) is an opt-in primal with storage. The default stays dense BroydenSolver(). No eviction reproduces its iterates up to roundoff; finite memory discards the oldest rank-one correction. The first step is Picard at unit init_scale, and damped Picard otherwise. solve gives the compact update and its limits.

LP-002’s “reverse-input” item means solve_input(...; ad): hold fixed and solve for , then differentiate that solution with respect to observed and parameters. The input derivative of solve_state was already covered. For the linear cell:

ForwardDiff, ChainRules/Zygote, Enzyme forward/reverse and Mooncake now implement this first-order IFT through the shared solve primitive, including dense and preconditioned Krylov solves. An internal problem adapter changes which residual coordinate is unknown. Warm starts, reports and returned Lux state have no tangent. The AD equation holds incoming state fixed and advances it once at the root. The existing no-AD reverse solve retains its legacy state threading, so stateful AD and no-AD solves need not agree. Branch selection remains the solver’s job; the derivative is local to its converged, nonsingular branch. Nonsquare systems, nonconvergence and singular residual Jacobians remain errors in AD.

Krylov accepts a left operator or a problem factory returning (; left, right); ldiv=false applies inverse preconditioners with mul!, and ldiv=true uses ldiv!. Factories receive a transposed flag and must honor it. Built-in JacobiIFT() probes the current operator diagonal in products and stores entries. on_solve records each RHS’s iteration count and actual residual. The full contract, setup costs and final residual policy are in solve.

For a scalar loss depending only on the forward fixed point, use deq_hvp(loss, f, x, ps, st, v; wrt=:ps, ad=AutoForwardDiff()). :x selects the input Hessian block and :joint includes input/parameter mixed blocks in input-then-parameter order. The result is a flat vector in the selected coordinates. A dedicated second-order adjoint differentiates residuals and loss twice at one root; no solver trajectory is differentiated. Only dense CPU real ForwardDiff residual/ loss derivatives are supported, with quadratic storage in the combined coordinates. The reverse polarity uses solve_input_hvp below. Krylov HVPs and arbitrary AD nesting remain unsupported. Unsupported backend selections and detected nested primitive derivatives throw ArgumentError; use the dedicated API.

test/solver_numerics.jl compares reverse derivatives to the closed form above, a nonlinear selected branch and reference differences. HVP tests compare to the analytic linear solve (including mixed blocks), scalar analytic second derivatives, and finite differences of first-order IFT gradients. See LP-002 for run measurements.

Batch A: reverse-input second order (2026-10-27)

solve_input_hvp(loss, f, z, ps, st, v; wrt=:ps, ad=AutoForwardDiff(), x₀=...) uses the same dense second-order adjoint through the input-problem adapter. :z selects observed state; :joint orders observed state before flattened parameters. The loss is a function of solved input only. Incoming state is fixed; branch selection uses x₀. Square dimensions, a converged root and nonsingular input residual Jacobian are required. Backend, dense-storage and nesting limits are identical to deq_hvp.

In the derivation in solve, replace the unknown by x and observed coordinate by z; the equations and signs are unchanged. test/lp002_batch_a.jl compares linear mixed Hessian blocks to the closed-form inverse and finite differences of first-order IFT gradients. For x=sqrt(z-a*z²-b-bias) it checks the analytic second state derivative and the full joint Hessian product. No root iterations are differentiated.

Device first-order rules (2026-10-27)

test/device_arrays.jl covers Picard, dense Broyden and limited Broyden roots in both polarities with JLArrays scalar indexing disabled. Device default warm starts follow the observed array’s storage and precision; explicit guesses still select their own storage. CPU defaults are unchanged. Both observed-channel and parameter IFT gradients match linear closed forms with opt-in KrylovIFT, including retained Broyden and Jacobi preconditioning. Active warm starts have zero derivative. DenseIFT has no working LU on JLArrays and gives a clear error; no host fallback is supplied. Enzyme/Mooncake device rules, device HVPs and complex rules are not supported. See solve for storage, tolerances and ForwardDiff directional costs.

Coupled Mycelium replay passes its selected AD backend to DEQ message solves via the DI extension’s private gradient-message hook. Both message directions now use IFT rules, checked against nonzero downstream linear losses on JLArrays. Raw ChainRules/Zygote inference replays the same primitive. Use the graph gradient entry point with explicit ad for ForwardDiff; raw implicit infer! is not an automatic backend-selection interface. Reverse-input message replay with nonempty Lux state throws, since legacy primal state threading and fixed-state IFT define different equations. Normal inference/default diagonal training are unchanged.

Raw ForwardDiff DEQ messages now reject dual-valued unannotated solves with an ArgumentError pointing to the explicit graph gradient driver, preventing accidental root-iteration differentiation. Ordinary primal messages are unchanged.

Trained by its residual with the state observed (2026-11-05, LP-022)

is not a density of : with observed, a cell that is the identity in satisfies every pair. test/lp022.jl (50 pairs of , a linear cell): at the generating relation, 0 at the identity cell, and the gradient in (0.62, equal to the closed form and to finite differences) points there. Mycelium.learns_unnormalised(f::DEQFactor, observed) is true when the state channel is observed, so free_energy_gradient warns; an observed input alone does not. Options: a residual with outside , a loss on the solved state, or the log-determinant that would make the factor a density of (not implemented). See Why Learned Relations Do Not Collapse.