SciML’s implicit layers as Lenticulum factors: the equilibrium family of Implicit Learners, alongside
VariationalDiffusion.jl’s diffusion family.
Sources: code:
ImplicitLayers.jl,deq.jl,flow.jl,luxfactor.jl,neuralode.jl,solve.jlTheory (CT-ML wiki): Bayesian Inversion · Statistical Game · Parametric Lens · Variational Free Energy · Lens
The thesis, in one table
DeepEquilibriumNetworks.jl and DiffEqFlux.jl both hand you an AbstractLuxLayer — a
function whose direction is fixed at construction. Both are built out of a relation. So
there are two ways to wrap them:
| wrapper | you supply | polarities | what you get |
|---|---|---|---|
LuxFactor | the assembled DeepEquilibriumNetwork / NeuralODE | 1 | graph membership; a Lux layer with extra steps |
DEQFactor | the cell | 2 (if square) | the relation, solvable either way |
NeuralODEFactor | the dynamics | 2, always | both endpoint integrations; numerical inverse depends on method |
The test suite makes the comparison on one object: the same cell wrapped as a LuxFactor has
one polarity and as a DEQFactor has two, with identical parameters.
The files
| file | note |
|---|---|
solve.jl | solve — Picard, dense/limited Broyden, SolveReport, preconditioned IFT |
deq.jl | deq — both solve polarities, first-order AD and dense HVP |
flow.jl | flow — fixed-step RK forwards and backwards, CNF divergence |
neuralode.jl | neuralode — the flow relation |
luxfactor.jl | luxfactor — any Lux layer, one polarity |
Concept notes: The Equilibrium Family, DEQ as a Relation, NeuralODE as an Invertible Factor.
Dependencies, and the AD line
There is no hard SciML dependency. The package depends on LenticulumCore, Mycelium,
LuxCore, Random, LinearAlgebra and DispatchDoctor. Built-in root solvers and Euler/RK4
remain available without SciML. Optional SciMLIntegrator loads through package extensions;
OrdinaryDiffEqTsit5 supplies Tsit5, OrdinaryDiffEqSDIRK supplies midpoint/trapezoid, and
both bring OrdinaryDiffEqCore and SciMLBase without the umbrella’s unrelated solvers.
See flow for loading, reversibility conditions, CNF augmentation and tests.
Optional AD now uses DifferentiationInterface with a user-supplied ADTypes backend.
Both DEQ solve polarities have first-order IFT rules for ForwardDiff, Zygote,
Enzyme and Mooncake, using dense LU by default or opt-in preconditioned Krylov.
LimitedBroydenSolver bounds primal storage. deq_hvp and solve_input_hvp provide dedicated dense
second-order adjoints using ForwardDiff residual and loss derivatives.
Euler/RK4 endpoint AD also supports runtime-activity Enzyme; SciMLSensitivity
supports tested Tsit5/Vern7/Kvaerno3/KenCarp4 endpoint algorithms with ForwardDiff/Zygote/Mooncake
and opt-in fixed nonempty Lux state; runtime-activity Enzyme reverse covers default Tsit5; see neuralode. The base
package still uses finite differences for derivative information:
RED-Diff’s stop-gradient means the diffusion family needs no derivative of the network at all, so avoiding AD costs it nothing. The equilibrium family’s backward pass is a linear system built from the Jacobian — the IFT is not an approximation that can be skipped. Avoiding AD here costs scalability:
fd_jacobianis forward passes and is honestly labelled a test-scale tool.
The base derivative tools are suitable for small systems. Optional endpoint sensitivities have the tested scope above; no general device support is established for this wrapper.
What it exposed
- Inversions still return points.
GaussianBeliefnow lives inLenticulumCore, but these factors still returnDiracBelief. A general non-Gaussian pushforward belief is not implemented;flow_logdetcomputes the continuous density correction without transporting a belief. See The Equilibrium Family §5. - Dirac-valued inversions do not have Bethe free energies. Both factors here contribute
energy and no entropy, so the counting correction of Bethe Free Energy has nothing to
correct and a mixed graph’s total is not . Same conclusion
VariationalDiffusionreached. - The DEQ’s singular Jacobian is the algebraic family’s discriminant. Two of the three Implicit Learners families fail in the same place for the same reason — see solve §3.
Related: Implicit Learners, The Equilibrium Family, Lux as a Parametric Lens, Factors are Parameterized Statistical Games
Batch A (2026-10-27) adds BroydenIFT() for per-solve retained inverse history,
with explicit transpose orientation, and state_policy=:fixed for constant-per-solve
flow state with one final update. Defaults remain DenseIFT, dense Broyden and
stage/evaluation state threading. solve, deq and flow give the tested
scope and unsupported combinations; test/lp002_batch_a.jl supplies the oracles.
JLArrays coverage (2026-10-27)
test/device_arrays.jl checks the device subset with scalar indexing disabled:
both DEQ polarities with three primal solvers and opt-in KrylovIFT, and Euler/RK4
flows/gradients. CPU defaults are retained. DenseIFT requires backend device LU,
which JLArrays lacks. Hardware GPUs, SciML device sensitivities, device second order
and Enzyme/Mooncake device rules are untested/unsupported; see solve and flow.