Entry point for the second of the three Implicit Learners families, implemented in
lib/ImplicitLayers.jl.The claim: SciML already builds these models out of relations, and then hands you a function. A
DeepEquilibriumNetworkis with one direction chosen and the solve sealed inside; aNeuralODEis a flow with the time direction fixed. Un-sealing them is what this package does, and the number of supported polarities is the measure of how much came back.
Sources: Bai, Kolter, Koltun, Deep Equilibrium Models, NeurIPS 2019; Chen, Rubanova, Bettencourt, Duvenaud, Neural Ordinary Differential Equations, NeurIPS 2018; Grathwohl et al., FFJORD, ICLR 2019; DeepEquilibriumNetworks.jl and DiffEqFlux.jl; Broyden, A class of methods for solving nonlinear simultaneous equations, 1965
Bibliography: Bai et al. 2019 · Chen et al. 2018 · Grathwohl et al. 2019 · Broyden 1965
Theory (CT-ML wiki): Hypergraph Category · Bayesian Inversion · Parametric Lens · Variational Free Energy · Lens
1. What SciML gives you
Both packages produce an AbstractLuxLayer:
DeepEquilibriumNetwork(cell, solver) # (x, ps, st) ↦ (z★, st) — solve info in st
NeuralODE(dynamics, tspan, Tsit5()) # (x, ps, st) ↦ (ODESolution, st)That is excellent engineering and it is a function. Lux as a Parametric Lens explains precisely what that costs: a Lux layer is a lens, so its direction is fixed at construction, and a Lenticulum factor only becomes a lens once a polarity is chosen. A wrapped SciML layer therefore has exactly one polarity.
The relation is still in there:
and is README’s definition of an implicit learner, verbatim. Keeping rather than the solved function is the whole intervention.
2. Three wrappers, and the number that distinguishes them
| wrapper | you supply | polarities | note |
|---|---|---|---|
LuxFactor | the assembled layer | 1 | works on anything; see luxfactor |
DEQFactor | the cell | 2 if , else 1 | DEQ as a Relation |
NeuralODEFactor | the dynamics | 2, unconditionally | NeuralODE as an Invertible Factor |
The test suite makes the comparison on a single object: the same cell, wrapped as a
LuxFactor, has one polarity; as a DEQFactor, two — same network, same parameter count.
That pair of numbers is the shortest statement of what the package is for.
LuxFactor is not a straw man. It buys graph membership, parameter management, channel names
and free-energy accounting, and it works today on every model in the SciML ecosystem with no
dependency on any of them. It just does not buy bidirectionality, and bidirectionality is what
the rest of the project is about.
3. The two reverse directions are not equally cheap
This is the finding worth carrying away.
A DEQ’s reverse direction is a hard root-find. Solving for is a general
nonlinear problem with no contraction structure to lean on, it needs to be
square at all, and convergence is an open question on every call. supports_polarity refuses
it when the shapes disagree — so a DEQ whose input is an embedding of a different width is
correctly reported as unidirectional and gains nothing here.
A NeuralODE’s reverse direction is another integration. The continuous flow is invertible
where solutions exist uniquely in both directions. Numerically, the same integrator runs
with the endpoints swapped. Built-in Euler/RK4 use equal stage counts, but do not invert the
discrete forward map exactly, even in exact arithmetic. They now use
SolverInversion(integrator). Adaptive methods may need different work in each direction.
The optional SciML extension adds Tsit5 and fixed-step symmetric methods such as midpoint
and trapezoid. Only a symmetric method on matching time grids, with converged stages,
restricted solver options and empty Lux state can receive ExactInversion, up to nonlinear
solve tolerance and roundoff. flow documents these conditions and tests.
Euler’s round trip is ; RK4 has fourth-order forward error and a fourth-order
round-trip bound, with a sharper fifth-order round trip on the scalar linear oracle.
examples/flow_roundtrip.jl and test/roundtrip.jl in ImplicitLayers measure and test these:
Euler doubling ratios tend to 2; linear RK4 round-trip ratios to 32; RK4 forward-error ratios
to 16. neuralode §4.3 gives the table and derivation. At , Euler’s endpoint
residual is 0.394994 and its inferred start is 20.9% off; these are not harmless roundoff.
What contracts forwards expands backwards. The inverse of , , amplifies absolute endpoint perturbations by , independently of discretisation. No solver removes that conditioning, and an accurate inverse of the numerical map need not recover the continuous initial point accurately. See NeuralODE as an Invertible Factor.
4. Where the derivatives come back
The Diffusion Family got away without automatic differentiation: RED-Diff’s stop-gradient means the denoiser’s Jacobian is never formed, so that package’s entire inversion is forward passes.
The equilibrium family cannot do that. Its backward pass is the implicit function theorem,
which is a linear system built from the Jacobian. There is no version of this that skips the
derivative. lib/ImplicitLayers.jl computes it by dense finite differences and labels that a
test-scale tool — forward passes where AD would be — so the package does
inference at any scale and sensitivity analysis only at small scale. See solve §4.1.
That is a real difference between the families and the vault had not previously noticed it: one family’s approximation buys it independence from AD; the other’s exactness costs it.
A cross-family connection
ift_sensitivity fails when is singular, and that is not a numerical
accident: there the implicit function theorem no longer guarantees a locally unique, smooth
fixed point. It does not by itself mean the relation branches ( has a unique real root
everywhere and is singular at ; DEQ as a Relation §4). The algebraic family calls the
singular locus the discriminant (Branches and the Discriminant). Two of the three
Implicit Learners families therefore fail at the same locus, in different vocabulary.
5. GaussianBelief was in the wrong package
Resolved (October 2026)
The recommendation below was carried out:
GaussianBeliefis inLenticulumCore, its rules inMycelium.VariationalDiffusionnow returns Laplace Gaussians (implicit_factor §5); the twoImplicitLayersrows below are still open, now as work rather than as a wall. The section is kept as the record of why.
Every inversion in lib/ImplicitLayers.jl returns a DiracBelief, because a root-find and an
ODE solve both produce a point. To return a distribution you would need somewhere to put
one — and GaussianBelief lives in the top-level Lenticulum package, which no lib/
package may depend on.
This is now the third factor package to hit it:
| package | what it can compute | what it must return |
|---|---|---|
VariationalDiffusion | RED-Diff’s variational (derived in the paper, dropped) | DiracBelief |
ImplicitLayers (DEQ) | — a covariance could be pushed through | DiracBelief |
ImplicitLayers (NeuralODE) | flow_logdet — the exact change-of-variables correction | DiracBelief |
The last row is the sharpest: flow_logdet computes precisely the quantity a density
transport needs, it is tested and correct, and it is dead code, because there is no belief
type in scope to hold the result.
The recommendation
GaussianBeliefand its canonical-form arithmetic belong inLenticulumCore, next toDiracBelief,SampleBeliefandTrivialBelief— which are already there. Nothing about it depends on the top-level package; it ended up there becauseLenticulumis where the first factor that needed it was written. Moving it would unblock Gaussian message passing in everylib/package at once.The same argument, from the other direction, is Acausal Composition is a Hypergraph Category §4: improper
GaussianBeliefs are what make the whole framework a hypergraph category. A load-bearing type should not be stranded at the top of the dependency graph.
6. Dirac-valued inversions have no Bethe free energy
Both factors here contribute an energy and no entropy term, because a
Dirac has none. So the counting correction of Bethe Free Energy — which adds back
copies of each variable’s entropy — has nothing to correct, and a graph mixing these
factors with a GaussianFactor produces a total that is not for any model.
The identity The Linear Gaussian Chain §4 verifies to machine precision simply stops holding. The Diffusion Factor §4.3 reached the same conclusion by a different route, and together they suggest the general statement:
A factor whose inversion returns a point cannot participate in a Bethe free energy. The formula is about entropies, and a point mass has none. Either the inversion returns a distribution (§5) or the factor is excluded from the accounting explicitly.
Sources
- Bai, Kolter, Koltun, Deep Equilibrium Models, NeurIPS 2019 — the fixed-point layer and the -memory IFT backward pass.
- Chen, Rubanova, Bettencourt, Duvenaud, Neural Ordinary Differential Equations, NeurIPS 2018 — the flow layer, the adjoint method, and the instantaneous change of variables.
- Grathwohl et al., FFJORD, ICLR 2019 — the Hutchinson trace estimator that makes the density correction affordable.
- DeepEquilibriumNetworks.jl and DiffEqFlux.jl — the Julia implementations being wrapped.
- Broyden, A class of methods for solving nonlinear simultaneous equations, 1965.
Related: Implicit Learners, DEQ as a Relation, NeuralODE as an Invertible Factor, The Diffusion Family, Lux as a Parametric Lens, Backpropagation by the Implicit Function Theorem, Bethe Free Energy, ImplicitLayers