ImplicitLayers
SciML's implicit layers — deep equilibrium networks and neural ODEs — as factors.
There is no hard SciML dependency: neither DeepEquilibriumNetworks.jl nor DiffEqFlux.jl is required, because what these wrappers need is the inner Lux network, not the assembled layer.
Three wrappers, and the difference matters
| wrapper | you supply | polarities |
|---|---|---|
LuxFactor | the assembled DeepEquilibriumNetwork / NeuralODE | 1 |
DEQFactor | the cell $g_\theta(z,x)$ | 2, if the residual is square |
NeuralODEFactor | the dynamics $f_\theta(z,t)$ | 2, always |
LuxFactor works today on anything in the SciML ecosystem and asks nothing of it. It gives you graph membership, parameter management and free-energy accounting — and one direction, because a Lux layer has already chosen which side is the input.
The other two keep the relation instead of the solved function, and get both directions:
# a DEQ: keep the residual r(x,z) = z - g(z,x)
f = DEQFactor(my_cell, (x = 4, z = 4); solver = BroydenSolver())
solve_state(f, x, ps, st) # solve for z — what SciML's DEQ does
solve_input(f, z, ps, st) # solve for x — what it cannot be askedThe reverse solve needs $\dim x = \dim z$; otherwise the residual is not square and supports_polarity refuses that direction rather than solving it badly.
Optional automatic differentiation
Load DifferentiationInterface, ADTypes and your backend package. For a DEQ solve, pass ad=AutoForwardDiff(), AutoZygote(), AutoEnzyme() or AutoMooncake() to solve_state or solve_input. Load ForwardDiff as well for the EnzymeCore/Mooncake extensions. The weak extensions implement the IFT, without differentiating root-finder iterations. Both input and parameter derivatives are tested against the linear-cell closed form and finite differences. Root convergence is required; singular residual Jacobians throw. Both solve polarities have first-order rules. Lux state is fixed throughout an AD root solve and advances once at the final root. Solve reports, initial guesses and state have no tangent.
deq_sensitivity(...; wrt=:ps, ad) gives the dense parameter sensitivity directly. DEQFactor(...; ift=DenseIFT()) defaults to dense LU. To use matrix-free IFT rules, load Krylov and select ift=KrylovIFT(rtol=1e-10, atol=1e-12, itmax=400, memory=20, precond=JacobiIFT()). Restarted GMRES uses only residual JVPs (forward) or VJPs (reverse). It throws on nonconvergence and checks the final residual explicitly. Krylov.jl provides the operator-product interface directly, without another operator package. All four backends are checked against the linear exact solution, including a 200-dimensional contractive example that forbids the dense state-Jacobian API. deq_sensitivity stays dense. The native Enzyme/Mooncake rules keep their existing inner ForwardDiff path in dense mode; in Krylov reverse mode each uses its own reverse backend. Hardware GPU execution remains untested; see the JLArrays subset below. Dedicated dense second-order products are described below. Enzyme outer closures capturing arrays need DI constant contexts or AutoEnzyme(function_annotation=Enzyme.Const); see solve.md for a reproducer.
Core's energy_jacobian(...; wrt=:ps, ad), energy_gradient and free_energy_gradient apply to DEQ, NeuralODE and scored Lux factors. Built-in Euler/RK4 endpoint flows use functional Lux state threading and differentiate the discrete map with ForwardDiff, Zygote, Mooncake and Enzyme. For Enzyme, pass AutoEnzyme(; mode = Enzyme.set_runtime_activity(Enzyme.Reverse)) (or Forward): constant model/stage memory flows through an active fold, requiring runtime activity. The factor derivative and graph training entry points check the mode before AD and raise an ArgumentError with that configuration when it is missing. Arbitrary outer DI calls must select it themselves. Plain AutoEnzyme() works for DEQ and scored LuxFactor paths. The ODE activity check does not require ForwardDiff; DEQ IFT does. neuralode.md describes the activity requirement. Tests derive both scalar discrete derivatives, check nonempty Lux state and fit a known ODE parameter using Mycelium's graph parameter_step. LuxFactor(...; dims=(nin,nout), residual=true) opts into squared endpoint-residual scoring; the default wrapper still delegates energy to downstream losses.
Preconditioning and second order
KrylovIFT(precond=M) uses a left operator. A factory precond=ctx -> (; left=M, right=N) can construct either side from the current problem; ctx.transposed distinguishes reverse solves. Omitted sides are identity. The default ldiv=false applies mul! with approximate inverses; ldiv=true applies ldiv!. Operators follow Krylov.jl's protocol. JacobiIFT() supplies an inverse diagonal using n operator probes and O(n) storage; small/nonfinite diagonal entries throw. on_solve=r -> ... receives a named tuple with iterations, residual and solved for each RHS. The final original residual is checked even with preconditioning. The built-in inverses require ldiv=false.
BroydenIFT() reuses the actual LimitedBroydenSolver forward inverse history as a left preconditioner, transposed for adjoint solves. Select DEQFactor(...; solver=LimitedBroydenSolver(), ift=KrylovIFT(precond=BroydenIFT())). History belongs to one solve; there is no shared cache or additional root solve. Other root solvers throw. A warm start may yield no secants, and limited memory can lose useful information; performance is problem dependent.
using ForwardDiff, DifferentiationInterface, ADTypes
hv = deq_hvp(z -> sum(abs2,z)/2, f, x, ps, st, v;
wrt=:ps, ad=AutoForwardDiff())This dedicated second-order adjoint computes a flat Hessian-vector product of loss(zstar) using dense residual and loss derivatives, without differentiating root iterations. wrt=:x selects inputs; :joint selects input followed by flattened parameters and includes mixed blocks. The loss may depend on input and parameters only through the solved state. Only DenseIFT(), CPU real floating point arrays and AutoForwardDiff() are supported. Storage is quadratic in the combined state/input/parameter dimension. Unsupported backend selections raise ArgumentError; arbitrary nesting of the first-order solve rules is unsupported. The tests compare linear products to the closed-form solve, nonlinear scalar second derivatives, and finite differences of first-order IFT gradients.
solve_input_hvp(loss, f, z, ps, st, v; wrt=:ps, ad=AutoForwardDiff(), x₀=...) extends the same dense adjoint to loss(xstar). wrt=:z selects observed state; :joint selects observed state followed by parameters, including mixed blocks. It has the same dense CPU/backend/loss restrictions as deq_hvp.
The no-AD solve_input retains legacy state threading. Its AD path holds incoming state fixed and advances it once at the root; stateful cells can therefore produce different roots in these two paths. Warm starts have zero derivative in both solve polarities' AD rules. Square dimensions alone do not guarantee a nonsingular root.
Neural ODE endpoint integration
The continuous flow has an inverse wherever solutions exist uniquely in both directions. Numerical reverse integration approximates that inverse; Euler/RK4 do not invert their discrete forward map, even in exact arithmetic. Both endpoint polarities are supported:
f = NeuralODEFactor(my_net, 4; tspan=(0.0,1.0), integrator=RK4Integrator())
flow_forward(f, z₀, ps, st)
flow_reverse(f, z₁, ps, st)
flow_logdet(f, z₀, ps, st)RK4Integrator(steps=50) remains the default. Euler's round-trip error is $O(h)$. RK4 has fourth-order forward error and an $O(h^4)$ round-trip bound for smooth dynamics; the scalar linear round trip is sharper, $O(h^5)$. Its doubling ratio is 32, while the forward-error ratio is 16. lib/ImplicitLayers.jl/examples/flow_roundtrip.jl reproduces the table in the implementation note: at $a=-5,T=1,N=50$, Euler's residual is 0.394994 with a 20.9% start-point error, RK4's is $6.953127\times10^{-7}$. At $N=1$ RK4's is 895.18. Inverting a contracting flow separately amplifies absolute endpoint perturbations by $e^{|a|T}$; a better integrator cannot remove this conditioning.
Optional OrdinaryDiffEq extension
Install the solver subpackages, then load them with ImplicitLayers:
using ImplicitLayers, OrdinaryDiffEqCore, OrdinaryDiffEqTsit5, OrdinaryDiffEqSDIRK
adaptive = SciMLIntegrator(Tsit5(); reltol=1e-9, abstol=1e-12)
fixed = SciMLIntegrator(ImplicitMidpoint(); adaptive=false, dt=0.02,
reltol=1e-12, abstol=1e-12)
f = NeuralODEFactor(my_net, 4; integrator=fixed)Core/SciMLBase supply the solve interface; Tsit5 and SDIRK supply the selected algorithms. These are weak dependencies, so the full OrdinaryDiffEq umbrella and its other solvers are unnecessary. The test environment resolves Core 4.19.0, Tsit5 2.1.5 and SDIRK 2.9.8.
The extension solves an out-of-place ODEProblem over either time direction and returns only the final state. It signs positive dt automatically, disables intermediate storage, and rejects failed/incomplete solves. CNF divergence is a finite-difference trace integrated as an augmented state; the built-in quadrature is unchanged. Lux state is threaded through every RHS evaluation, including rejected stages and divergence probes.
inversion_kind gives Euler/RK4 and adaptive Tsit5 SolverInversion(integrator). Midpoint and Trapezoid are symmetric methods (Hairer–Lubich–Wanner II.3); with adaptive=false, matching grids (dt divides the interval), a default step limiter, and only dt, adaptive, abstol, reltol, maxiters options, the trait permits ExactInversion up to nonlinear-solve tolerance and roundoff. assemble also requires empty Lux state. This describes the discrete inverse, not exactness against the continuous ODE. Tests use abstol=reltol=1e-12 and check relative round-trip residual below $10^{-10}$.
Load SciMLSensitivity to enable SciML endpoint parameter AD:
using SciMLSensitivity, OrdinaryDiffEqTsit5
itg = SciMLIntegrator(Tsit5(); abstol=1e-12, reltol=1e-10)
# Optional continuous adjoint:
adjoint_itg = SciMLIntegrator(Tsit5(); abstol=1e-12, reltol=1e-10,
sensealg=GaussAdjoint(autojacvec=ZygoteVJP()))
f = NeuralODEFactor(my_net, 4; integrator=itg)
J = energy_jacobian(f, z0, nothing, y, ps, st; wrt=:ps, ad=AutoZygote())The default sensealg=nothing resolves to ForwardDiffSensitivity() for small parameter counts, as recommended by the SciMLSensitivity manual. Flattened Lux parameters are passed as ODEProblem.p, with a zero-valued shape template in the RHS. With the default state_policy=:threaded, the sensitivity path requires recursively empty incoming/returned Lux state. Nonempty-state primal evaluation and reference differences preserve their previous threading; factor/graph AD rejects that combination before tracing.
Opt into NeuralODEFactor(...; state_policy=:fixed) for functional nonempty state. All RHS evaluations use incoming state as a constant; after successful integration, a single evaluation at final endpoint/time produces returned state. Euler/RK4 implement the same opt-in policy, while their default continues stage threading. State itself is not differentiated. In-place state mutation and fixed-state flow_logdet are unsupported.
Tests cover Tsit5, Vern7 (OrdinaryDiffEqVerner) and stiff Kvaerno3/KenCarp4 (OrdinaryDiffEqSDIRK), in both directions, with default sensitivity, InterpolatingAdjoint and GaussAdjoint (ZygoteVJP). ForwardDiff, Zygote and Mooncake gradients are compared to affine closed forms and fixed-state RK4. The reverse Kvaerno3/InterpolatingAdjoint pair fails its closed-form oracle on the tested stack and throws at factor/graph AD entry points. Use GaussAdjoint, KenCarp4, or direct AutoForwardDiff instead. Primal evaluation stays available; examples/lp002_sciml_adjoint_limit.jl exposes the failing gradient directly. Runtime-activity Enzyme reverse scalar loss gradients are tested for default Tsit5 sensitivities: AutoEnzyme(mode=Enzyme.set_runtime_activity(Enzyme.Reverse)). Vector-return Enzyme split-reverse Jacobians are unsupported (energy_jacobian throws; use the other backends for Jacobians). Enzyme forward fails in OrdinaryDiffEq callback initialization on the tested stack and gets a configuration error at factor/graph entry points. examples/lp002_enzyme_limits.jl reproduces this and the outer-closure annotation boundary. No Enzyme support is claimed for other solver/sensealg combinations. Arbitrary outer DI calls must enforce these restrictions themselves.
Sensitivity selection leaves inversion labels and reversal semantics unchanged. CNF/divergence AD, callbacks, GPU and higher flow derivatives are untested. Continuous adjoints can retain intermediate trajectories internally. Backsolve reverse-conditioning concerns still apply.
Solvers
PicardSolver is the naive "keep applying the layer" loop and converges only if the iteration is a contraction. BroydenSolver is derivative-free quasi-Newton and converges on plenty of problems where Picard diverges — it is the default, and it is what DeepEquilibriumNetworks.jl reaches for too.
LimitedBroydenSolver(memory=20) opts into good inverse Broyden with at most 20 rank-one corrections, O(memory*n) storage and work, and no dense inverse. With no eviction its iterates agree with dense Broyden up to roundoff. The first step is Picard for init_scale=1, damped Picard otherwise. Dropping old corrections can slow convergence; there is no line search.
All root solvers return a SolveReport (converged, iterations, residual, function evaluations) rather than throwing. A solver that stopped early is an inexact inversion, which is legal.
ift_sensitivity implements the implicit function theorem, $\partial z^\ast/\partial x = (I - \partial_z g)^{-1}\partial_x g$ — one linear solve, no unrolling.
Known gaps
- Dense IFT remains the default. Jacobi setup requires one operator probe per coordinate.
- SciML support is limited to the solver/backend/state-policy combinations tested above; divergence AD and Enzyme forward endpoint AD remain open.
- Second-order IFT is limited to the dedicated dense adjoints in both polarities above.
- Built-in explicit methods have no error estimate and may diverge on coarse or stiff problems. The optional solvers do not remove inverse conditioning or establish GPU support.
- Every inversion returns a
DiracBelief.
API
ImplicitLayers.ImplicitLayersImplicitLayers.AbstractIntegratorImplicitLayers.AbstractRootSolverImplicitLayers.BroydenIFTImplicitLayers.BroydenSolverImplicitLayers.DEQFactorImplicitLayers.DEQModelImplicitLayers.DenseIFTImplicitLayers.EulerIntegratorImplicitLayers.JacobiIFTImplicitLayers.KrylovIFTImplicitLayers.LimitedBroydenSolverImplicitLayers.LuxFactorImplicitLayers.NeuralODEFactorImplicitLayers.NeuralODEModelImplicitLayers.PicardSolverImplicitLayers.RK4IntegratorImplicitLayers.SciMLIntegratorImplicitLayers.SolveReportImplicitLayers._divergenceImplicitLayers.apply_layerImplicitLayers.cell_valueImplicitLayers.default_cell_inputImplicitLayers.default_dynamics_inputImplicitLayers.deq_hvpImplicitLayers.deq_sensitivityImplicitLayers.fd_jacobianImplicitLayers.flow_forwardImplicitLayers.flow_logdetImplicitLayers.ift_sensitivityImplicitLayers.integrateImplicitLayers.integrate_with_divergenceImplicitLayers.inversion_kindImplicitLayers.isreversibleImplicitLayers.residualImplicitLayers.residualImplicitLayers.solve_inputImplicitLayers.solve_input_hvpImplicitLayers.solve_rootImplicitLayers.solve_stateImplicitLayers.vectorfield
ImplicitLayers.ImplicitLayers — Module
ImplicitLayersSciML's implicit layers — deep equilibrium networks and neural ODEs — as Lenticulum factors. The equilibrium family of Implicit Learners.md, alongside VariationalDiffusion.jl's diffusion family.
The thesis
DeepEquilibriumNetworks.jl and DiffEqFlux.jl both hand you an AbstractLuxLayer: a function $x \mapsto y$ whose direction is fixed when you construct it. But both are built out of a relation —
\[\text{DEQ:}\quad z = g_\theta(z,x) \qquad\qquad \text{NeuralODE:}\quad \frac{dz}{dt} = f_\theta(z,t)\]
— and the layer is that relation with one direction chosen and the solve sealed inside. So there are two ways to wrap them, and the difference is the whole point of this package:
| wrapper | what you give it | polarities | what you get |
|---|---|---|---|
LuxFactor | the assembled DeepEquilibriumNetwork / NeuralODE | 1 | graph membership; a Lux layer with extra steps |
DEQFactor | the cell $g_\theta$ | 2 (if square) | the relation; solvable for either channel |
NeuralODEFactor | the dynamics $f_\theta$ | 2, always | both endpoint integrations; exactness depends on method |
LuxFactor works today on anything in the SciML ecosystem and asks nothing of it. The other two need the inner network rather than the assembled layer, and give back the bidirectionality the rest of the project is about.
The files
| file | note | supplies |
|---|---|---|
solve.jl | [[solve]] | Picard, dense/limited Broyden, SolveReport, preconditioned IFT |
deq.jl | [[deq]] | DEQFactor — both solve polarities, first-order AD and dense HVP |
flow.jl | [[flow]] | fixed-step RK, forwards and backwards, CNF divergence |
neuralode.jl | [[neuralode]] | NeuralODEFactor — the flow relation |
luxfactor.jl | [[luxfactor]] | LuxFactor — any Lux layer, one polarity |
Concept notes: [[The Equilibrium Family]], [[DEQ as a Relation]], [[NeuralODE as an Invertible Factor]].
Two properties worth knowing up front
No hard SciML dependency. Built-in root solvers and Euler/RK4 work without SciML. Optional SciMLIntegrator uses OrdinaryDiffEqCore/SciMLBase and solver subpackages such as OrdinaryDiffEqTsit5 and OrdinaryDiffEqSDIRK. Explicit time reversal only approximates the inverse of the numerical forward map; symmetric fixed-step methods on matching grids can invert it up to nonlinear-solve tolerance. See inversion_kind. Optional DI derivatives accept a user-selected ADTypes backend. DEQ solves have IFT rules in weak ChainRulesCore, ForwardDiff, EnzymeCore and Mooncake extensions. Euler/RK4 endpoint flows use functional state threading with ForwardDiff, Zygote and Mooncake, plus Enzyme with runtime activity enabled, for example AutoEnzyme(; mode = Enzyme.set_runtime_activity(Enzyme.Reverse)) (or Forward). Plain AutoEnzyme() remains supported for DEQ and scored LuxFactor paths. Loading SciMLSensitivity enables SciML endpoint derivatives with ForwardDiff/Zygote/Mooncake and recursively empty Lux state. DEQFactor(...; ift=KrylovIFT()) opts into matrix-free first-order solves via the Krylov weak extension. Base derivative tools use finite differences and remain small-system tools; see solve.md §4.
Every inversion returns a DiracBelief. A root-find and an ODE solve both produce a point. GaussianBelief is now in LenticulumCore, so a Gaussian answer (a covariance pushed through deq_sensitivity) is possible but not built yet. See [[The Equilibrium Family]] §5.
ImplicitLayers.AbstractIntegrator — Type
abstract type AbstractIntegratorA numerical ODE scheme. Built-in Euler/RK4 are fixed-step explicit methods; optional SciMLIntegrator supports adaptive and implicit solvers.
ImplicitLayers.AbstractRootSolver — Type
abstract type AbstractRootSolverA method for solving $F(u) = 0$ given only the ability to evaluate $F$.
Derivative-freeness is a requirement, not a convenience: the residual of a DEQ contains a neural network, and this package has no automatic-differentiation dependency (see ImplicitLayers.md). It is also what DeepEquilibriumNetworks.jl does in practice — Broyden and limited-memory Broyden are its workhorses.
ImplicitLayers.BroydenIFT — Type
BroydenIFT()Opt-in KrylovIFT(precond=BroydenIFT()) left inverse preconditioner using the actual forward LimitedBroydenSolver history. Uses its transpose for adjoint solves. Requires ldiv=false. History is owned by one AD solve, never global; no extra root solve or dense Jacobian is used. Warm starts may leave no secants. Up to two original-residual correction solves handle rescaled recursive residuals; iteration diagnostics include corrections, sharing any explicit itmax budget.
ImplicitLayers.BroydenSolver — Type
BroydenSolver(; maxiters = 100, tol = 1e-10, init_scale = 1.0)"Good" Broyden: a quasi-Newton method maintaining an approximate inverse Jacobian updated by the Sherman–Morrison formula from secant pairs.
Derivative-free, and it is what DeepEquilibriumNetworks.jl reaches for. Note the initialisation: with $H_0 = I$ the first step is
\[u_1 = u_0 - H_0F(u_0) = u_0 - (u_0 - g(u_0)) = g(u_0)\]
— the first Broyden step is exactly a Picard step, and everything after it is the correction Picard never makes. Asserted in the test suite.
Stores a dense $n\times n$ matrix, so it is $O(n^2)$ in memory. Real DEQs use limited-memory Broyden for exactly this reason; see solve.md §4.2.
ImplicitLayers.DEQFactor — Type
DEQFactor(cell, dims::NamedTuple; solver = BroydenSolver(), channels = (:x, :z),
input = default_cell_input, ift = DenseIFT())A deep equilibrium model as a relation: the fixed-point condition $z = g_\theta(z, x)$ presented as a residual, solvable for either channel.
cell is any LuxCore.AbstractLuxLayer computing $g_\theta(z,x)$ — including a DeepEquilibriumNetworks.jl cell, which is exactly such a layer. dims gives the two channel dimensions. Parameters and state are the cell's, untouched. ift=KrylovIFT() opts into matrix-free GMRES in solved-state AD rules (load Krylov and DI); dense LU is the default.
f = DEQFactor(my_cell, (x = 4, z = 4); solver = BroydenSolver(maxiters = 200))Solving $z = g(z,x)$ for x is $\dim z$ equations in $\dim x$ unknowns. Unless $\dim x = \dim z$ it is over- or under-determined and the root solve is not asking a well-posed question. LenticulumCore.supports_polarity refuses that polarity rather than solving it badly; see deq.md §3.
Contrast LuxFactor, which wraps an already assembled DeepEquilibriumNetwork and gets one polarity — a Lux layer with extra steps.
ImplicitLayers.DEQModel — Type
DEQModel(factor, polarity)The open model the polarity selects: "solve this fixed-point equation for the unobserved channel". Pure — a deterministic relation with no internal randomness.
ImplicitLayers.DenseIFT — Type
Dense residual Jacobian and LU (the default first-order IFT rule).
ImplicitLayers.EulerIntegrator — Type
EulerIntegrator(steps = 50)Explicit Euler. First order, and present mainly as the thing RK4Integrator is checked against — its error is visible enough to make the order of accuracy testable.
ImplicitLayers.JacobiIFT — Type
JacobiIFT(; min_diagonal=1e-14)Opt-in KrylovIFT(precond=JacobiIFT()) inverse diagonal preconditioner. Uses n operator probes and O(n) storage, without forming the state Jacobian. Rejects nonfinite or small diagonal entries. Uses mul! (ldiv=false).
ImplicitLayers.KrylovIFT — Type
KrylovIFT(; rtol=1e-10, atol=1e-12, itmax=0, memory=20,
precond=nothing, ldiv=false, on_solve=nothing)Opt-in matrix-free GMRES IFT solve. Load Krylov and DifferentiationInterface. itmax=0 uses Krylov's default (twice the state dimension). Restart after memory iterations to bound basis storage. Failure to meet the residual tolerance throws.
precond accepts a left operator or a factory ctx -> (; left=M, right=N); omitted sides are identity. ctx contains operator, transposed, factor, solved (channel name), joint (observed channel followed by parameters), root, template, offsets and state. Build preconditioners for the indicated orientation. Krylov applies mul! by default (operators approximate the inverse); ldiv=true applies ldiv! instead. JacobiIFT() probes the operator diagonal. on_solve, when supplied, receives (iterations, residual, solved) as a named tuple after each linear solve, including failures; residual is measured in the original, unpreconditioned system.
ImplicitLayers.LimitedBroydenSolver — Type
LimitedBroydenSolver(; memory=20, maxiters=100, tol=1e-10, init_scale=1.0)Good inverse Broyden with at most memory rank-one secant corrections to init_scale*I. Stores two vectors per correction, never a dense inverse: O(memory*n) storage and work per step. Evicts the oldest correction without rebuilding the others. With no eviction, reproduces dense Broyden up to roundoff. The first step is damped Picard with damping init_scale (ordinary Picard when it is one).
ImplicitLayers.LuxFactor — Type
LuxFactor(layer, :in => :out; dims = (nothing, nothing), postprocess = identity, residual = false)Any LuxCore.AbstractLuxLayer as a unidirectional factor.
The point of least resistance for reusing the SciML ecosystem: a DeepEquilibriumNetwork, a NeuralODE, a Chain, a Boltz vision backbone all satisfy the interface, so all of them wrap without this package knowing anything about them.
postprocess adapts the layer's output to a plain array — needed for the SciML layers whose call returns a solution object rather than a vector:
# DiffEqFlux's NeuralODE returns an ODESolution
LuxFactor(NeuralODE(net, (0.0, 1.0), Tsit5()), :x => :y;
postprocess = sol -> Array(sol)[:, end])
# DeepEquilibriumNetworks returns the steady state directly, and reports the solve in `st`
LuxFactor(DeepEquilibriumNetwork(cell, NewtonRaphson()), :x => :z)Set residual=true (with an explicit output dimension in dims) to score the endpoint vector y - layer(x) with a squared norm. Core parameter derivatives and Mycelium's fixed-message parameter_step can then train this factor. The default keeps zero energy for compatibility with downstream-loss graphs. Direct losses on apply_layer can also be differentiated with LenticulumCore.parameter_gradient.
isunidirectional(::LuxFactor) == true. A Lux layer is a lens ([[Lux as a Parametric Lens]]); a factor only becomes one once a polarity is chosen, and a layer has already chosen. So this wrapper buys graph membership, parameter management and free-energy accounting — and none of the bidirectionality the rest of the project is about. Use it when the model genuinely is a function; use DEQFactor or NeuralODEFactor when it is not.
ImplicitLayers.NeuralODEFactor — Type
NeuralODEFactor(dynamics, dim; tspan = (0.0, 1.0), integrator = RK4Integrator(),
channels = (:z0, :z1), input = default_dynamics_input, state_policy=:threaded)A neural ODE as a bidirectional factor: the relation $\{(z_0,z_1) : z_1 = \Phi_{t_0\to t_1}(z_0)\}$ where $\Phi$ is the flow of $dz/dt = f_\theta(z,t)$.
state_policy=:threaded advances Lux state at each stage (the default). Opt-in :fixed holds incoming state constant during a solve, then advances it once by evaluating dynamics at the final endpoint/time. State is not a learnable coordinate; functional updates are required. Fixed-state logdet is unsupported.
dynamics is any LuxCore.AbstractLuxLayer computing $f_\theta(z,t)$ — the same object you would hand to DiffEqFlux.NeuralODE. Parameters and state are its own.
f = NeuralODEFactor(my_net, 4; tspan = (0.0, 1.0), integrator = RK4Integrator(steps = 100))Both channels have dimension dim — a flow cannot change the dimension of its state, which is why this factor takes one dim rather than two and why a NeuralODE cannot be used as an encoder that compresses. Contrast DEQFactor, whose two channels are independent.
NeuralODE(model, tspan, Tsit5()) is a Lux layer: (n)(x, ps, st) builds an ODEProblem and returns (ODESolution, st). Direction fixed at construction, output a solution object. Wrapping that gives you LuxFactor and one polarity; this factor keeps the flow and gets two.
ImplicitLayers.NeuralODEModel — Type
NeuralODEModel(factor, polarity)The open model the polarity selects. Both directions integrate the dynamics; the inversion label depends on the numerical method, time grid and deterministic dynamics assumption.
ImplicitLayers.PicardSolver — Type
PicardSolver(; maxiters = 100, tol = 1e-10, damping = 1.0)Damped fixed-point iteration $u \leftarrow u - \beta F(u)$.
For the DEQ residual $F(z) = z - g(z,x)$ this is exactly $z \leftarrow (1-\beta)z + \beta g(z,x)$: the naive "just keep applying the layer" loop.
A contraction is sufficient for convergence, and nothing checks that in advance; it is not a necessary condition for a particular initial guess. That is the caveat Implicit Learners.md records — "this only works if the iteration converges, and unconstrained DEQs need not" — and the test suite exhibits a linear cell with spectral radius > 1 on which this solver diverges and BroydenSolver does not.
ImplicitLayers.RK4Integrator — Type
RK4Integrator(steps = 50)Classical fourth-order Runge–Kutta, fixed step. The default: for the smooth non-stiff dynamics a NeuralODE usually has, steps = 50 of RK4 is far more accurate than anything Euler reaches, and it is Tsit5's spiritual ancestor without the adaptivity.
ImplicitLayers.SciMLIntegrator — Type
SciMLIntegrator(alg; sensealg = nothing, kwargs...)Optional OrdinaryDiffEq solver, loaded by using OrdinaryDiffEqCore and a solver package (e.g. OrdinaryDiffEqTsit5 or OrdinaryDiffEqSDIRK). Keywords go to solve; only the final state is returned (explicit adjoints may retain a trajectory internally). dt is a positive magnitude, signed automatically for reverse solves.
Fixed-step ImplicitMidpoint/Trapezoid can invert their discrete map up to nonlinear-solve tolerance; adaptive solvers such as Tsit5 only control integration error. See inversion_kind for the conservative exactness conditions. Loading SciMLSensitivity enables endpoint derivatives with recursively empty Lux state, or functional nonempty state under NeuralODEFactor(...; state_policy=:fixed). sensealg=nothing selects ForwardDiffSensitivity() for small problems; pass an explicit adjoint to override it. Sensitivity options do not affect inversion labels.
ImplicitLayers.SolveReport — Type
SolveReport(converged, iterations, residual, nfe)What a root solve is willing to say about itself.
nfe (number of function evaluations) is reported because it is the honest cost measure for an implicit layer — DeepEquilibriumNetworks.jl reports it too, in its DeepEquilibriumSolution, for the same reason: iteration count means nothing when one Broyden step and one Picard step cost differently.
[!important]
converged == falseis not an errorBayesian Lens.md: a solver that stopped early is an inexact inversion, and inexact inversions are legal — the free energy records the cost. So the report is threaded out rather than thrown, and it is the caller's business what to do about it.
ImplicitLayers._divergence — Method
_divergence(vf, u, t) -> Real$\operatorname{tr}(\partial \mathrm{vf}/\partial u)$ by forward differences.
ImplicitLayers.apply_layer — Method
apply_layer(f::LuxFactor, x, ps, st) -> (y, st)One forward pass, with postprocess applied to the output.
ImplicitLayers.cell_value — Method
cell_value(f, z, x, ps, st) -> (g, st)One forward pass of the cell: $g_\theta(z, x)$.
ImplicitLayers.default_cell_input — Method
default_cell_input(z, x) = (z, x)How a DEQ cell is called: LuxCore.apply(cell, input(z, x), ps, st). Override via the input keyword for cells expecting a different arrangement — the factor deliberately knows nothing about the cell's internals.
ImplicitLayers.default_dynamics_input — Method
default_dynamics_input(z, t) = (z, t)How the dynamics layer is called: LuxCore.apply(dynamics, input(z, t), ps, st).
ImplicitLayers.deq_hvp — Method
deq_hvp(loss, f, x, ps, st, direction; wrt=:ps, ad, z₀=zeros(statedim(f)))Dense second-order IFT adjoint for the scalar loss(z★). Returns a flat Hessian-vector product in core parameter order (wrt=:ps), input order (:x), or input followed by parameters (:joint). Load ForwardDiff and DI and pass ad=AutoForwardDiff(). Differentiates residuals and loss twice, never solver iterations. Requires DenseIFT(), a converged root, and a nonsingular Jacobian. CPU real arrays only; loss may depend on x/parameters only through z★. Arbitrary nesting of the first-order solve rules is unsupported: use this API.
ImplicitLayers.deq_sensitivity — Method
deq_sensitivity(f, x, z★, ps, st; wrt = :x, ad = nothing)The dense IFT sensitivity -(D_z r) \ D_u r at a supplied root. wrt=:x selects inputs; wrt=:ps selects flattened parameters in the core tree order. With ad=nothing, use central differences; an ADTypes object uses the optional DI extension. The incoming state is fixed, and the root must have a nonsingular residual Jacobian. For differentiating a scalar loss through the root itself use solve_state(...; ad); its weak AD rules avoid differentiating solver iterations.
ImplicitLayers.fd_jacobian — Method
fd_jacobian(F, u; ε) -> MatrixThe dense Jacobian $\partial F/\partial u$ by forward differences: $n+1$ evaluations of F.
This remains a reference for small systems. Optional DI derivatives and solved-state IFT rules use automatic differentiation instead; see deq_sensitivity and solve_state(...; ad). The root iterations themselves do not need derivatives.
Column $j$ uses the relative step ε (default $\max(10^{-7}, \sqrt{\varepsilon_T})$ for element type $T$) and divides by the increment the rounded probe actually makes.
ImplicitLayers.flow_forward — Method
flow_forward(f, z₀, ps, st) -> (z₁, st)
flow_reverse(f, z₁, ps, st) -> (z₀, st)Integrate $t_0 \to t_1$ and $t_1 \to t_0$ respectively. Same integrator, endpoints swapped — that is the entire implementation of the reverse direction.
ImplicitLayers.flow_logdet — Method
flow_logdet(f, z₀, ps, st) -> (z₁, Δlogdet, st)Forward flow together with $\int \operatorname{tr}\partial_z f_\theta\,dt$, so that
\[\log p(z_1) = \log p(z_0) - \Delta\!\log\!\det\]
the instantaneous change of variables. This is the piece that would make the factor transport a density rather than a point. It is not used for that yet: GaussianBelief is now reachable (in LenticulumCore), but a flow needs a non-Gaussian, pushforward belief, which does not exist. See neuralode.md §4.2.
ImplicitLayers.ift_sensitivity — Method
ift_sensitivity(Jz, Jx) -> MatrixThe implicit function theorem, applied to a fixed point $z^\ast = g(z^\ast, x)$:
\[(I - \partial_z g)\,\frac{\partial z^\ast}{\partial x} = \partial_x g \qquad\Longrightarrow\qquad \frac{\partial z^\ast}{\partial x} = (I - \partial_z g)^{-1}\,\partial_x g\]
One linear solve, no unrolling — storage independent of the root iteration count (the dense state Jacobian still takes quadratic storage), and the reason Implicit Learners.md lists the IFT as the equilibrium family's backward pass. See [[Backpropagation by the Implicit Function Theorem]].
Throws if $I - \partial_z g$ is singular: the IFT does not establish a locally differentiable inverse there. This alone does not imply nonuniqueness (a cubic residual can have a unique root with zero derivative). Dense matrix reference only.
ImplicitLayers.integrate — Method
integrate(vf, u₀, t₀, t₁, integrator) -> u₁Integrate $du/dt = \mathrm{vf}(u, t)$ from t₀ to t₁.
t₁ < t₀ is allowed and is the whole point: dt is simply negative and the same scheme runs backwards. vf is called as vf(u, t).
ImplicitLayers.integrate_with_divergence — Method
integrate_with_divergence(vf, u₀, t₀, t₁, integrator) -> (u₁, Δlogdet)Integrate the state and the log-density correction of a continuous normalising flow:
\[\frac{d}{dt}\log p(u(t)) = -\operatorname{tr}\frac{\partial \mathrm{vf}}{\partial u} \qquad\Longrightarrow\qquad \log p(u_1) = \log p(u_0) - \underbrace{\int_{t_0}^{t_1}\!\operatorname{tr}\,\partial_u \mathrm{vf}\,dt}_{\texttt{Δlogdet}}\]
This is the instantaneous change-of-variables of Chen et al. / FFJORD, and it is what turns a NeuralODE from a map on points into a map on densities — i.e. what would let this factor push a real belief rather than a Dirac.
[!warning] The trace is computed by a dense finite-difference Jacobian $O(n)$ extra evaluations of
vfper divergence evaluation. Built-in methods use trapezoidal quadrature; SciML integrates an augmented state. FFJORD's Hutchinson estimator avoids the dense trace; seeflow.md§4.1. What is here is correct, checkable against a linear system, and unusable above smalln.
ImplicitLayers.inversion_kind — Method
inversion_kind(integrator, t₀, t₁)SolverInversion(integrator) by default. ExactInversion() is reserved for a symmetric fixed-step method whose positive dt divides the interval (within floating-point roundoff). This is exactness of the discrete map, up to roundoff and nonlinear-solve tolerance, not accuracy against the continuous solution. It assumes dynamics do not change with Lux state.
ImplicitLayers.isreversible — Method
isreversible(integrator) -> BoolWhether the configured numerical method is symmetric under time reversal, assuming a deterministic vector field and converged implicit stages. Euler/RK4 and adaptive solvers return false. Symmetry alone does not ensure the two solves use matching time grids.
ImplicitLayers.residual — Method
residual(f::DEQFactor, x, z, ps, st) -> (r, st)$r = z - g_\theta(z,x)$, the vector energy. Membership of the relation is $r \approx 0$, which is [[README]]'s definition of an implicit learner instantiated exactly.
ImplicitLayers.residual — Method
residual(f::NeuralODEFactor, z₀, z₁, ps, st) -> (r, st)$r = z_1 - \Phi_{t_0\to t_1}(z_0)$, the vector energy.
ImplicitLayers.solve_input — Method
solve_input(f, z, ps, st; x₀, ad=nothing) -> (x★, SolveReport, st)Solve $z = g_\theta(z, x)$ for x, holding z fixed — the direction a DeepEquilibriumNetwork cannot be asked for.
Requires $\dim x = \dim z$. Even then it is a general nonlinear root-find in x with no contraction structure to lean on, so Broyden is the only sensible solver and convergence is a genuinely open question per call. That is why the SolveReport is returned rather than discarded.
With ad, supports the same first-order IFT backends as solve_state, requires convergence, holds incoming Lux state fixed and advances it once at the root. Without ad, retains legacy state threading across residual evaluations.
ImplicitLayers.solve_input_hvp — Method
solve_input_hvp(loss, f, z, ps, st, direction; wrt=:ps, ad, x₀=zeros(indim(f)))Dense second-order IFT adjoint for loss(x★) in the reverse polarity. wrt=:z selects the observed state, :ps flattened parameters, and :joint state followed by parameters. Holds incoming Lux state fixed and advances it once at the root, as solve_input(...; ad). The restrictions of deq_hvp apply.
ImplicitLayers.solve_root — Method
solve_root(F, u₀, solver) -> (u, SolveReport)Solve $F(u) = 0$ from the initial guess u₀.
F is called as F(u) and must return something the same shape as u. The residual norm that decides convergence is the Euclidean norm; tol is absolute, which is a simplification worth knowing about (solve.md §4.3).
ImplicitLayers.solve_state — Method
solve_state(f, x, ps, st; z₀, ad = nothing) -> (z★, SolveReport, st)Solve $z = g_\theta(z, x)$ for z. The forward DEQ pass — what DeepEquilibriumNetworks.jl does.
z₀ defaults to zeros, which is SkipDeepEquilibriumNetwork's point of departure: a learned initial guess can improve convergence and is not implemented (deq.md §4.4).
The incoming Lux state is fixed throughout the iterations, then advanced once at the returned root. Supply an ADTypes backend to use the optional IFT solve rules (tested with ForwardDiff, Zygote, Enzyme and Mooncake; the latter two also need ForwardDiff loaded). In this AD path a nonconverged solve throws; reports and returned state are bookkeeping without tangents. solve_input(...; ad) supports the same first-order backends in the other polarity.
ImplicitLayers.vectorfield — Method
vectorfield(f, ps, st) -> (vf, stref)The dynamics as a plain closure vf(z, t), plus the Ref accumulating the threaded Lux state.
The Ref is a wart with a cause: integrate wants a pure (u,t) -> du function, while LuxCore wants st threaded in and out of every call. Something has to hold the state across the integrator's inner calls, and a Ref is the smallest thing that does. See neuralode.md §4.1.
LenticulumCore.assemble — Method
LenticulumCore.assemble(f::NeuralODEFactor, p, ps, st)Select the inversion label with inversion_kind(f.integrator, f.tspan...). Euler/RK4 and adaptive SciML methods use SolverInversion(integrator): reversing time approximates the continuous inverse, not the inverse of the discrete residual map. Symmetric fixed-step SciML methods on matching grids use ExactInversion, up to nonlinear solve tolerance and roundoff. Nonempty Lux state conservatively keeps SolverInversion, since state updates can change the dynamics. See neuralode.md §4.3 for convergence and conditioning limitations.
LenticulumCore.free_energy — Method
LenticulumCore.free_energy(lens, π, inputs, ps, st)For an assembled DEQ/NeuralODE lens, evaluate the residual energy at invert's point, then add the prior's anchor cost on the unobserved channel. inputs is a Dirac observation or a NamedTuple containing that observation on the polarity's observed channel.
The anchor is zero for a trivial prior; zero at an exactly equal Dirac point and infinite elsewhere; or ½(u-μ)'Λ(u-μ) for a proper Gaussian. The Gaussian cost is relative to its mean (no log-normalization constant). Channel precisions do not soften a Dirac prior. Soft observations and other beliefs throw. Entropy uses the zero point-belief convention.
This is Definition 20's expectation under the inversion, with this specified residual and anchor energy. assemble(f, p) restricts the relation to one polarity; it does not turn the root/flow inversion into a posterior conditioned on the anchor. See point_game.md.
LenticulumCore.pointwise_energyspace — Method
LenticulumCore.pointwise_energyspace(::DEQFactor)$E_c = \mathbb{R}^{\dim z}$: the residual itself, as a vector.
This is what [[Scalar and Multivariate Energy]] asks for and what VariationalDiffusion's factor could not provide — here the residual is a genuine $\mathbb{R}^n$-valued object and the natural scalarisation is $\tfrac12\|\cdot\|^2$. Keeping it vector-valued is exactly what the implicit function theorem needs, per that note's §6.3.
LenticulumCore.supported_polarities — Method
LenticulumCore.supported_polarities(f::DEQFactor)Both directions when $\dim x = \dim z$; only the forward one otherwise.
The count is the point. A DeepEquilibriumNetwork has one direction, fixed when you construct it. This has two whenever the residual is square, and the second one is a genuinely new capability rather than a re-labelling — it solves the same equation for the other variable.
LenticulumCore.supported_polarities — Method
LenticulumCore.supported_polarities(f::NeuralODEFactor)Both endpoint polarities are supported. Each runs the configured integrator with the appropriate endpoints. Built-in fixed-step methods use equal work in both directions, but reverse integration need not invert the discrete forward map. Adaptive/implicit solvers may also require different work. Contracting flows amplify perturbations when inverted.
Mycelium.factor_message — Method
Mycelium.factor_message(f::LuxFactor, target, polarity, inputs, prior, ps, st)A DiracBelief on the output channel. Asking for a message on the input channel throws a PolarityError-shaped ArgumentError, because that is precisely the direction a Lux layer does not have.
Mycelium.factor_message — Method
Mycelium.factor_message(f::DEQFactor, target, polarity, inputs, prior, ps, st)The factor → variable message: a DiracBelief on target, from a root solve.
The incoming prior is used as the solver's initial guess and nothing else. That is a genuinely good use for it — warm-starting from the current belief is what makes iterated message passing over implicit layers cheap — and it is not the same as conditioning on it. This point site does not condition on the prior; see deq.md §4.2.
Mycelium.factor_message — Method
Mycelium.factor_message(f::NeuralODEFactor, target, polarity, inputs, prior, ps, st)A DiracBelief on target, obtained by integrating in the appropriate direction.
The prior is genuinely unused here — unlike DEQFactor, where it warm-starts the solver. A flow has nothing to warm-start.
Mycelium.local_free_energy — Method
Mycelium.local_free_energy(f::DEQFactor, msgs, ps, st)$\tfrac12\|r\|^2$ at the incoming messages' points, or 0.0 when a channel carries no point.
Note what this is not: there is no entropy term, because the inversion returns a Dirac and a Dirac has none. So this factor contributes energy only, and the Bethe counting correction of [[Bethe Free Energy]] has nothing to correct here. deq.md §4.3.
Mycelium.local_free_energy — Method
Mycelium.local_free_energy(f::NeuralODEFactor, msgs, ps, st)$\tfrac12\|z_1 - \Phi(z_0)\|^2$ when both endpoints carry points, else 0.0.
This is zero when the end point was produced by the same numerical forward map. Reverse-solved endpoints can have nonzero energy from round-trip error. The energy measures endpoint consistency with the numerical map; it cannot establish continuous-flow accuracy.
Game quantities and conformance
The default LuxFactor requires a Dirac input observation y and contributes zero game energy and entropy. With residual=true, its game instead scores Dirac endpoints in the first/second channel order, with zero point-belief entropy. DEQFactor and NeuralODEFactor implement two games:
free_energy(f, π, y, ps, st)scores incoming Dirac points:πon the first channel (xor start),yon the second (zor end), following GaussianFactor's fixed order. It equalsMycelium.local_free_energyfor those messages. Entropy is scalar zero; non-Dirac arguments throw.lens = first(assemble(f, p, ps, st))restricts the relation to one polarity.free_energy(lens, π, inputs, ps, st)runs that inversion and scores its returned point: residual energy plus an anchor on the unobserved channel.inputscontains a named Dirac observation (or that Dirac directly). A trivial prior costs zero; an incompatible hard Dirac prior costs infinity; a proper Gaussian costs $\tfrac12(u^*-\mu)^\top\Lambda(u^*-\mu)$ relative to its mean, without a normalization constant. The inversion still uses the prior only as a warm start (DEQ), or ignores it (ODE). Soft observations, singular Gaussian anchors and other prior families throw. Channel precision does not soften Dirac priors.
The lens game is Definition 20's expectation under the point inversion, with zero point-belief entropy. Both games return (F, st) in ScalarEnergySpace(); scalar_free_energy also accepts these lenses and a scalarisation override σ. Scoring arbitrary points uses energy/scalar_energy (L-ADR-0003), not a free_energy keyword.
If the residual is affine in the unknown and vanishes at the solved point, the incoming energy equals the solved residual energy plus a Gaussian anchor with precision $J_u^\top J_u$ centered at the incoming unknown. energy_jacobian supplies this metric. The tests derive 1/8 for g(z,x)=z/2+x at x=1,z=3 in both directions, and 2 for zero ODE dynamics at endpoints 1,3 in both directions. A nonlinear quadratic DEQ checks cubic absolute energy error for the first-order Gauss–Newton approximation. The common factor checker now covers the complete point game for both factors without LP-009 exemptions.
Solved-state IFT AD accepts AutoForwardDiff(), AutoZygote(), AutoEnzyme() and AutoMooncake(). Load DI, the backend and (for Enzyme/Mooncake) ForwardDiff; missing extensions and other solve backends are rejected before solving. Pass the backend to solve_state and the outer derivative. Enzyme outer closures that capture arrays need DI constant contexts or function_annotation=Enzyme.Const; see the implementation note solve.md for the limitation and reproducer. Hardware GPU execution remains untested; see the JLArrays subset below. Second order is limited to the dedicated dense adjoint above.
Device arrays (LP-002 batch B)
JLArrays tests run with scalar indexing disabled in Float32 and Float64. Picard, Broyden and LimitedBroyden solve both DEQ polarities. First-order ForwardDiff/Zygote IFT gradients use KrylovIFT, optionally with Jacobi, retained Broyden or supplied left/right preconditioners. Choose root/linear tolerances appropriate for Float32. Device default guesses and solver storage follow the observed array; CPU defaults are unchanged. DenseIFT requires device LU, unavailable on JLArrays, and gives an ArgumentError recommending KrylovIFT. No dense CPU fallback is hidden in that rule.
Euler/RK4 forward/reverse endpoints, input/parameter derivatives and parameter energy/free-energy/graph entry points are tested against discrete closed forms and CPU. Device time coefficients preserve Float32; both Lux state policies are tested with empty and functionally updated state. Time-dependent fixtures check stage and final state-update time precision against an augmented linear-system oracle. A ChainRules loop helper propagates threaded-flow Zygote input cotangents correctly while retaining the original fold primal for native backends.
Core's ForwardDiff device adapter uses DI directional pushforwards; native ForwardDiff vector seeding still scalar-indexes. Cost grows with active parameter count. Hardware GPU execution, SciML devices, divergence, device HVPs and Enzyme/Mooncake device rules are outside this coverage. The latter rules and device reference differences give explicit unsupported-combination errors.
Divergence sensitivity boundary (LP-002 batch C)
Endpoint sensitivity does not automatically cover the CNF density correction. AD through integrate_with_divergence must differentiate the trace's explicit parameter dependence and its trajectory dependence. The original trace is a mutable, dense forward-difference computation; AD of those probes, when it works, gives the finite-probe derivative. Stateful probe calls also advance the default Lux evaluation state, while state_policy=:fixed still rejects logdet. Adaptive-SciML, device and general native-backend divergence sensitivities remain open. examples/lp002_metric_divergence_limits.jl isolates raw trace, integrated linear correction and Sampson metric localisation against independent closed forms. LP-002 records the observed backend results and remaining work.
Graph-level realised-schedule and converged-message gradients are Mycelium APIs; see Mycelium. The new implicit graph adjoint has no dependency on ImplicitLayers and currently requires empty message state and CPU numeric Gaussian/Dirac coordinates. Factor-level IFT and the graph fixed-point equation are separate differentiation problems.