implementation

SciML’s implicit layers as Lenticulum factors: the equilibrium family of Implicit Learners, alongside VariationalDiffusion.jl’s diffusion family.

Sources: code: ImplicitLayers.jl, deq.jl, flow.jl, luxfactor.jl, neuralode.jl, solve.jl

Theory (CT-ML wiki): Bayesian Inversion · Statistical Game · Parametric Lens · Variational Free Energy · Lens

The thesis, in one table

DeepEquilibriumNetworks.jl and DiffEqFlux.jl both hand you an AbstractLuxLayer — a function whose direction is fixed at construction. Both are built out of a relation. So there are two ways to wrap them:

wrapperyou supplypolaritieswhat you get
LuxFactorthe assembled DeepEquilibriumNetwork / NeuralODE1graph membership; a Lux layer with extra steps
DEQFactorthe cell 2 (if square)the relation, solvable either way
NeuralODEFactorthe dynamics 2, alwaysboth endpoint integrations; numerical inverse depends on method

The test suite makes the comparison on one object: the same cell wrapped as a LuxFactor has one polarity and as a DEQFactor has two, with identical parameters.

The files

filenote
solve.jlsolve — Picard, dense/limited Broyden, SolveReport, preconditioned IFT
deq.jldeq — both solve polarities, first-order AD and dense HVP
flow.jlflow — fixed-step RK forwards and backwards, CNF divergence
neuralode.jlneuralode — the flow relation
luxfactor.jlluxfactor — any Lux layer, one polarity

Concept notes: The Equilibrium Family, DEQ as a Relation, NeuralODE as an Invertible Factor.

Dependencies, and the AD line

There is no hard SciML dependency. The package depends on LenticulumCore, Mycelium, LuxCore, Random, LinearAlgebra and DispatchDoctor. Built-in root solvers and Euler/RK4 remain available without SciML. Optional SciMLIntegrator loads through package extensions; OrdinaryDiffEqTsit5 supplies Tsit5, OrdinaryDiffEqSDIRK supplies midpoint/trapezoid, and both bring OrdinaryDiffEqCore and SciMLBase without the umbrella’s unrelated solvers. See flow for loading, reversibility conditions, CNF augmentation and tests.

Optional AD now uses DifferentiationInterface with a user-supplied ADTypes backend. Both DEQ solve polarities have first-order IFT rules for ForwardDiff, Zygote, Enzyme and Mooncake, using dense LU by default or opt-in preconditioned Krylov. LimitedBroydenSolver bounds primal storage. deq_hvp and solve_input_hvp provide dedicated dense second-order adjoints using ForwardDiff residual and loss derivatives. Euler/RK4 endpoint AD also supports runtime-activity Enzyme; SciMLSensitivity supports tested Tsit5/Vern7/Kvaerno3/KenCarp4 endpoint algorithms with ForwardDiff/Zygote/Mooncake and opt-in fixed nonempty Lux state; runtime-activity Enzyme reverse covers default Tsit5; see neuralode. The base package still uses finite differences for derivative information:

RED-Diff’s stop-gradient means the diffusion family needs no derivative of the network at all, so avoiding AD costs it nothing. The equilibrium family’s backward pass is a linear system built from the Jacobian — the IFT is not an approximation that can be skipped. Avoiding AD here costs scalability: fd_jacobian is forward passes and is honestly labelled a test-scale tool.

The base derivative tools are suitable for small systems. Optional endpoint sensitivities have the tested scope above; no general device support is established for this wrapper.

What it exposed

  • Inversions still return points. GaussianBelief now lives in LenticulumCore, but these factors still return DiracBelief. A general non-Gaussian pushforward belief is not implemented; flow_logdet computes the continuous density correction without transporting a belief. See The Equilibrium Family §5.
  • Dirac-valued inversions do not have Bethe free energies. Both factors here contribute energy and no entropy, so the counting correction of Bethe Free Energy has nothing to correct and a mixed graph’s total is not . Same conclusion VariationalDiffusion reached.
  • The DEQ’s singular Jacobian is the algebraic family’s discriminant. Two of the three Implicit Learners families fail in the same place for the same reason — see solve §3.

Related: Implicit Learners, The Equilibrium Family, Lux as a Parametric Lens, Factors are Parameterized Statistical Games

Batch A (2026-10-27) adds BroydenIFT() for per-solve retained inverse history, with explicit transpose orientation, and state_policy=:fixed for constant-per-solve flow state with one final update. Defaults remain DenseIFT, dense Broyden and stage/evaluation state threading. solve, deq and flow give the tested scope and unsupported combinations; test/lp002_batch_a.jl supplies the oracles.

JLArrays coverage (2026-10-27)

test/device_arrays.jl checks the device subset with scalar indexing disabled: both DEQ polarities with three primal solvers and opt-in KrylovIFT, and Euler/RK4 flows/gradients. CPU defaults are retained. DenseIFT requires backend device LU, which JLArrays lacks. Hardware GPUs, SciML device sensitivities, device second order and Enzyme/Mooncake device rules are untested/unsupported; see solve and flow.