One root-finder serves both directions of a DEQ, because both directions are the same residual solved for a different variable. That is the entire reason a DEQ is worth expressing as a factor.
Sources: Montoison and Orban (2023), Krylov.jl, p. 1 (operator products); Saad and Schultz (1986), GMRES; code:
solve.jl,deq.jl,../ext/ImplicitLayersEnzymeCoreExt.jl,../ext/ImplicitLayersMooncakeExt.jl,../test/parameter_ad.jl,../test/solver_numerics.jlBibliography: Montoison & Orban 2023 · Saad & Schultz 1986
Theory (CT-ML wiki): Bayesian Inversion · Bayesian Lens · Lens
1. Root solvers and the convergence caveat
For the DEQ residual , Picard is “keep applying the layer”: . A contraction is sufficient for convergence, which is Implicit Learners’s recorded caveat — “this only works if the iteration converges, and unconstrained DEQs need not” — and the test suite makes it concrete: a linear cell with has a perfectly good unique fixed point that Picard cannot reach and Broyden reaches in a handful of steps.
Broyden is initialised with , which makes its first step
exactly a Picard step, with everything after it the correction Picard never makes.
Asserted in the tests when init_scale=1; other scales give damped Picard.
This equality says nothing about later convergence.
All three root algorithms are derivative-free, so inference has no hard AD dependency.
Optional AD of a solved state is a separate IFT calculation. These are also what
DeepEquilibriumNetworks.jl does — Broyden and limited-memory Broyden are its workhorses.
2. SolveReport, and why nothing throws
A non-convergent solve returns converged = false and the caller decides. That is
Inversions and Bayesian Lenses’s position quoted directly: a solver that stopped early is simply an
inexact inversion, and the loss records the cost. Throwing would make a legal-but-poor
inversion into an error.
nfe is reported alongside iterations because iteration count is meaningless when a
Broyden step and a Picard step cost differently — DeepEquilibriumNetworks.jl reports it in
its DeepEquilibriumSolution for the same reason.
3. The implicit function theorem
One linear solve, no unrolling — the property that makes implicit layers in memory and the reason Implicit Learners lists the IFT as this family’s backward pass. See Backpropagation by the Implicit Function Theorem.
ift_sensitivity throws SingularException when is singular, and that
deserves emphasis:
A singular prevents applying the IFT
It means the IFT does not establish a locally differentiable inverse; the relation may branch there, but uniqueness alone is not ruled out. The algebraic family calls that locus the discriminant (Branches and the Discriminant); it is literally the same phenomenon in a different model class, and it produces the same failure. Two of the three Implicit Learners families therefore fail in the same place for the same reason, which the vault had not previously connected.
4. Implementation difficulties
4.1 Optional AD uses the implicit function theorem
fd_jacobian remains a forward-difference reference. deq_sensitivity(...; wrt=:ps, ad) and the differentiable solve_state(...; ad) use optional DI derivatives instead.
For a residual they solve
The ChainRulesCore weak extension supplies an rrule for the internal solved-state
primitive. A cotangent gives by , then
via a backend VJP. The ForwardDiff weak extension
strips duals before the root solve, obtains residual JVPs in their seed directions,
and solves the same linear system to rebuild the root’s duals. Neither differentiates
Broyden/Picard iterations. Both the input and parameters are active; the initial guess,
solve report and returned Lux state are nondifferentiated bookkeeping.
AD requires a converged root and a nonsingular residual Jacobian. A singular Jacobian
means the IFT gives no local inverse; it does not prove nonuniqueness (for example
has a unique root and zero derivative there). The default DenseIFT() solve is dense: a reverse
backend VJP in the ChainRules rule can avoid parameter-Jacobian storage, but the
Enzyme/Mooncake rules use a dense ForwardDiff VJP. The state Jacobian and its
factorisation cost quadratic storage and cubic factorisation work. An opt-in KrylovIFT() path is described below. These primitive rules are first order. solve_input(...; ad) now reuses them in
the other polarity. A dedicated dense second-order adjoint is described below;
GPU and additional backends are untested.
The EnzymeCore extension provides native EnzymeRules.forward, augmented_primal
and reverse rules on _implicit_state. DI’s default AutoEnzyme() Jacobian uses
batched forward mode; its scalar gradient uses reverse mode. The Mooncake extension
registers a native reverse rrule!! with @is_primitive; it accumulates the joint
vector cotangent in Mooncake’s mutable forward data. Neither uses @from_rrule.
Both extensions also require ForwardDiff to be loaded. In dense mode they use AutoForwardDiff()
through DI for the residual state Jacobian, JVPs and VJPs: these evaluate only
_joint_residual, never the solve primitive. deq_sensitivity uses this same inner
backend for Enzyme/Mooncake. ForwardDiff VJPs here form a dense residual Jacobian;
this is a small-system reference, not a matrix-free implementation. The reverse
rules factor the transposed state Jacobian once per primal solve.
Only the joint vector is active. _solve_metadata makes the backend, model,
parameter template, state and initial guess constant to Enzyme. The template supplies
shapes, not parameter values. _joint_vector copies inputs and flattened parameters
separately: tracing vcat(constant_input, active_parameters) merges their activities
in Julia’s copy loop and fails Enzyme’s static activity analysis. ChainRules supplies
the corresponding slicing pullback for this packing helper. Array parameter leaves
are materialised before packing to avoid array/view union types. ForwardDiff strips
duals from initial guesses even when only the initial guess is active.
test/parameter_ad.jl checks all four backends against the linear-cell closed form
and central differences, including active warm starts, a zero warm-start-only
Jacobian and a nonzero downstream scalar loss. Nonlinear scalar parameters,
nonempty Lux state, endpoint gradients and graph parameter steps have separate checks.
4.2 Dense and limited-memory Broyden
The default BroydenSolver still stores an inverse. Opt-in
LimitedBroydenSolver(memory=m) stores transformed secant pairs :
Here , and is init_scale.
This is the same good inverse Broyden update as the dense solver. Products with
and cost ; no dense inverse is allocated. The absolute
1e-14 denominator guard matches the default solver. At capacity the oldest
rank-one correction is removed, without recomputing the retained corrections.
These are transformed secant pairs, not a multisecant interpolation of the last
raw pairs: eviction need not preserve the latest secant equation and can
slow convergence. There is no line search or global convergence guarantee.
Without eviction, iterates agree with dense Broyden up to roundoff; arithmetic
order differs, so bitwise equality is not claimed. The first step is
, exactly damped Picard (ordinary Picard for ).
SolveReport and nonconvergence behavior match the existing solvers. Tests cover
full iteration traces, finite memory, degenerate updates and nonfinite residuals.
Anderson acceleration is likewise absent, and is the other solver DEQ implementations commonly use.
4.3 The tolerance is absolute
tol compares against with no relative component and no scaling by . On
a badly scaled problem that is either unreachable or trivially satisfied. A real solver takes
abstol and reltol; NonlinearSolve.jl does, and if this package ever grows a dependency
it should be that one.
4.4 Root state is fixed during solve_state
A fixed-point solve must evaluate one fixed residual, including its incoming Lux state.
solve_state now holds that state fixed through the root iterations and advances it once
at the final root. This also defines the equation used by the IFT. State is not an
additional implicit variable. A cell that mutates arrays in its incoming state in place
violates this functional contract. solve_input(...; ad) uses this same contract;
without ad, it retains legacy Ref state threading to preserve the existing API.
For stateful cells these are deliberately different equations, so the legacy
trajectory is not a derivative oracle for the AD solve.
Related: deq, flow, neuralode, The Equilibrium Family, Backpropagation by the Implicit Function Theorem, Branches and the Discriminant
Solved-state IFT AD accepts AutoForwardDiff(), AutoZygote(), AutoEnzyme() and
AutoMooncake(). Plain AutoEnzyme() works for these DEQ rules without runtime
activity; the built-in ODE stage fold requires an explicit runtime-activity mode
(see neuralode). Other solve backends are rejected explicitly. Missing Enzyme/Mooncake
extensions also raise an error before the solve; load DI, the backend and ForwardDiff.
Pass the backend to solve_state as well as the outer derivative.
For Enzyme outer functions that capture arrays, use DI constant contexts (as the tests
do), or explicitly set AutoEnzyme(function_annotation=Enzyme.Const). Automatic
closure activity can raise EnzymeMutabilityException on this stack; it is outside
the solve rule. This reproduces with the linear-cell fixture from test/runtests.jl:
using Enzyme, ADTypes, DifferentiationInterface, ForwardDiff
function closure_probe(f, x, ps, st)
θ, off = LenticulumCore._flatten(ps), LenticulumCore.offset_tree(ps)
ad = AutoEnzyme()
loss = θ -> first(solve_state(f, x,
LenticulumCore._unflatten(θ, off, ps), st; ad))
DifferentiationInterface.jacobian(loss, ad, θ) # EnzymeMutabilityException
end
# f = DEQFactor(CELL, (x=2,z=2)); ps, st = LuxCore.setup(Xoshiro(3), f)
# closure_probe(f, [0.7, -0.3], ps, st)Validation including the runtime-activity flow follow-up (2026-10-26, Julia 1.13.1, CPU):
julia --project=<package> -e 'using Pkg; @time Pkg.test()' passed with the following
counts (sum of top-level test summaries) and Pkg.test() wall times:
| Package | Passed | Time |
|---|---|---|
| ImplicitLayers | 995 | 255.1 s |
| LenticulumCore | 316 | 46.8 s |
| Mycelium | 494 | 11.2 s |
The implicit environment resolved DI 0.7.21, Enzyme 0.13.216, EnzymeCore 0.8.26,
Mooncake 0.5.65 and ForwardDiff 1.4.6. The solve tests also exercise explicitly
selected Enzyme forward gradients and reverse Jacobians, besides DI’s default modes.
DEQ graph learning holds messages fixed; it differentiates endpoint residuals, not
message schedules. Its final solve_state check is a primal validation of the learned
parameter, separate from the solved-state derivative oracles.
Both runtime-activity Enzyme modes pass the Euler/RK4 and ODE graph learning checks;
plain AutoEnzyme() still passes DEQ and scored LuxFactor checks. The separate
activity extension requires only DI and EnzymeCore. The vault build passes with
246 HTML pages and no KaTeX errors; julia --project=docs docs/make.jl passes in
435.9 s (including tutorial execution and the vault). It reports size warnings for
the core API page and search index. Both builds use a disposable QUARTZ_DIR copy
with its checkout/clean EXIT hook removed.
Matrix-free first-order IFT (LP-002 follow-up)
using Krylov, DifferentiationInterface
f = DEQFactor(cell, (x=200, z=200); solver=PicardSolver(tol=1e-12),
ift=KrylovIFT(rtol=1e-10, atol=1e-12, itmax=400, memory=20))
z, report, stnew = solve_state(f, x, ps, st; ad)The primal root solver and its convergence policy are unchanged. The IFT option
selects only the linear solve inside the AD rules. For , forward rules
solve using residual JVPs. Reverse rules solve
with residual VJPs and return .
ChainRules uses the supplied reverse backend; the native Enzyme and Mooncake
rules use their own reverse backend in Krylov mode. Enzyme’s inner VJPs enable
runtime activity and a constant function annotation to handle mixed constant/active
cell inputs; users can still select plain AutoEnzyme() for the outer DEQ rule. ForwardDiff remains the inner
JVP backend for the Enzyme forward rule. Dense mode retains its existing
ForwardDiff linearisation for native Enzyme/Mooncake rules.
Krylov.jl is a weak dependency instead of the broader LinearSolve interface:
GMRES needs only a typed size/eltype/mul! operator, so no operator package or
matrix materialisation is needed. The operator has no indexing or conversion to a
matrix. Prepared DI products are reused across GMRES iterations. Restarted GMRES
stores O(memory) basis vectors (plus its small Hessenberg workspace).
Preconditioning is opt-in; see below. itmax=0 means Krylov’s default, twice
the state dimension. Nonconvergence throws ArgumentError, including when the
explicit final residual exceeds atol + rtol*norm(rhs) despite an optimistic
recursive estimate. Zero right-hand sides still undergo the residual check.
test/lp002_followups.jl checks all four existing backends plus explicit Enzyme
forward/reverse modes on the two-dimensional linear oracle, including forced
nonconvergence. A 200-dimensional random contractive cell checks scalar gradients
against and overrides _root_jacobian to throw if called.
This guards the state-Jacobian entry point independently of AD chunk size.
deq_sensitivity and ift_sensitivity remain dense reference APIs; selecting
Krylov on the factor changes differentiable solve_state, not those APIs.
Select Picard or LimitedBroydenSolver for a primal without a dense inverse.
GPU and complex parameters remain untested; second order has the dedicated scope below.
Solver numerics follow-up (2026-10-27)
Preconditioning and measurement
KrylovIFT(precond=...) accepts a left operator or a factory returning
(; left=M, right=N); omitted sides are identity. A factory receives the current
operator, transposed, factor, solved (channel name), joint, root, template, offsets, and
incoming state. The joint vector is observed channel then parameters, in either
polarity. Operators must implement Krylov’s mul! protocol, or ldiv! when
ldiv=true. With the default ldiv=false, supply approximate inverses: GMRES
operates on and recovers the unknown through . Factories are responsible
for orientation, including transposing their preconditioners for reverse solves.
Preconditioner construction is bookkeeping, without parameter derivatives.
JacobiIFT() supplies an inverse diagonal on the left. It probes the current
operator with each coordinate basis vector: JVPs in forward mode or VJPs
in reverse mode, storage, no matrix. This setup can be expensive and is
not a stochastic diagonal estimator. Nonfinite or small diagonal entries throw;
ldiv=true with this inverse preconditioner also throws. No forward Broyden
history is reused: doing so would require retaining solver workspace in the
primitive and defining its orientation for both polarities.
on_solve = report -> ... receives iterations, actual residual and solved status
for each RHS. Convergence still requires the original unpreconditioned residual
bound, not just Krylov’s recursive estimate. A preconditioner can change the
recursive stopping scale; the explicit final check can therefore reject an
apparently converged preconditioned solve. There is no automatic refinement.
The defaults remain DenseIFT() and BroydenSolver().
Reverse-input first-order IFT
LP-002’s history and deq describe “different variable held fixed.” We interpret
“reverse-input solve rules” as differentiating solve_input with observed ,
not as the input cotangent of solve_state (which was already tested).
For with square nonsingular :
An internal problem adapter swaps the residual coordinate solved by the existing
primitive. Thus ForwardDiff, ChainRules/Zygote, native Enzyme forward/reverse and
Mooncake use the same rules, dense or Krylov. No duplicated AD rules trace the
root iterations. The initial guess has zero derivative, including an active
warm start. Nonconverged roots and singular residual Jacobians fail as in
solve_state. Square shape alone does not ensure local invertibility.
Dedicated second-order adjoint
deq_hvp(loss, f, x, ps, st, direction; wrt=:ps, ad=AutoForwardDiff())
returns a flat Hessian-vector product for loss(zstar). :x selects the input
block; :joint selects input followed by flattened parameters and includes mixed
blocks. The loss has no direct input/parameter dependence. DenseIFT() and CPU
real floating-point arrays are required. Other backend selections are rejected
with ArgumentError. This is an explicit second-order API, not an arbitrary
nested derivative rule for the solve primitive.
Write , , , and . At fixed compute
ForwardDiff builds the residual Jacobian, the scalar contracted residual Hessian and the loss Hessian, evaluated at one converged root. Only residuals and loss are differentiated twice; the root finder is never traced. This reference uses quadratic storage in state plus input plus parameter dimension and dense solves; it is unsuitable for large parameter trees. State and warm start are fixed and carry no tangent. The HVP API itself has no higher-order rule.
ForwardDiff nesting of the solve primitive and ForwardDiff over the other three solve backends are rejected. The ChainRules pullback also has an explicit nested reverse guard. Use the dedicated adjoint instead of composing backend tapes; other native mixed/nested Enzyme/Mooncake tapes are outside the supported API.
test/solver_numerics.jl records iteration counts and checks linear full-trace
Broyden agreement, reverse-input closed forms and nonlinear branches, fixed state,
left/right and inverse/solve preconditioner conventions, and HVPs against an
analytic linear solution, scalar second derivatives and central differences of
first-order IFT gradients. Measurements and full verification are recorded in
LP-002 The Jacobian side is not implemented.
Batch A: retained Broyden history (2026-10-27)
KrylovIFT(precond=BroydenIFT()) opts into the actual forward
LimitedBroydenSolver inverse history. Each solve owns its correction vectors;
there is no global cache and no repeat solve. A private report carries history
across the AD primitive; the public return remains SolveReport. Every backend
passes that history into linear preparation. Repeated pullbacks only read it.
For H = scale*I + sum(aᵢ*bᵢ'), the left inverse preconditioner applies H in
forward solves and H’ in transposed solves (swap the two vector lists). This
also handles reverse polarity: history approximates the inverse of the residual
Jacobian in the actually solved coordinate, including its sign. It requires
LimitedBroydenSolver and ldiv=false; unsupported choices throw. A root warm
start may yield no secants, leaving just the initial scaled identity. Eviction
or poor secant information can reduce quality; no speedup is guaranteed. The
unpreconditioned final residual check remains mandatory. If the retained inverse
causes recursive residual convergence before the original tolerance is met, up
to two residual-correction solves refine the result. They share any explicit
itmax budget; on_solve.iterations includes every correction iteration. Failure
after refinement still throws. This refinement is opt-in through BroydenIFT and
leaves other preconditioners/defaults unchanged.
test/lp002_batch_a.jl measures GMRES counts on a rotated, nonsymmetric,
ill-conditioned linear problem, with no preconditioner, Jacobi and retained
Broyden. It checks gradients against DenseIFT on four backends, explicitly
checks H/H’ application, and checks both solve polarities. Counts exclude
forward root work and Jacobi setup; they do not establish a wall-time speedup.
Outer Enzyme closure boundary
examples/lp002_enzyme_limits.jl reproduces EnzymeMutabilityException for a
DI outer closure capturing arrays, then succeeds with
AutoEnzyme(function_annotation=Enzyme.Const). The implicit rule runs only after
Enzyme has selected the outer function’s activity. It cannot retroactively
annotate that closure. Automatically changing arbitrary DI calls would require
intercepting methods owned by DI/Enzyme, and treating every captured array as
constant could discard intentionally active closure data. No such global rule
is installed. Library-owned derivatives already put their callable/context in
DI.Constant; callers of arbitrary outer DI functions must choose annotation
or explicit constant contexts themselves. This remains an upstream/API boundary,
with the working explicit annotation tested in test/lp002_batch_a.jl.
Device arrays: LP-002 batch B (2026-10-27)
The optional GPUArraysCore extension keeps primal vectors, dense Broyden’s inverse,
and limited Broyden’s secant vectors on the input device. Its scales/damping are
converted to the vector element type. CPU allocations/scales retain their previous
behavior, including their Float64 promotion. No root algorithm or denominator guard
changes. A missing device initial guess is allocated with similar(observed, n);
CPU defaults remain Float64 zeros. Retained history is owned by the same solve.
Jacobi setup forms basis vectors by broadcast and reads diagonal slices without
host scalar indexing. GMRES tolerances are converted to its RHS real element type:
the previous Float64 keywords caused a type error with Float32 RHSs. User left/right
inverse operators, left ldiv=true, Jacobi and retained Broyden preconditioning are
checked on nonsymmetric two-dimensional residuals. The original residual check and
convergence failures remain in force. Very tight default tolerances need loosening
for Float32; the API does not silently change a user’s convergence criteria.
test/device_arrays.jl uses JLArrays with allowscalar(false), Float32/Float64,
closed-form linear roots and IFT products. Both solve polarities are covered with
Zygote and ForwardDiff. ForwardDiff dual reconstruction broadcasts over whole
columns instead of scalar-indexing each root component. Core’s device ForwardDiff
entry points assemble directional DI pushforwards, avoiding native vector seeding.
This costs one direction per active coordinate and is a small-problem reference,
not a claim of scalable forward parameter differentiation.
JLArrays has no device LU on the tested stack. DenseIFT attempts the backend factorisation and reports an ArgumentError pointing to KrylovIFT on failure; it never copies a Jacobian to the CPU. DenseIFT success on hardware GPUs is untested. Enzyme/Mooncake device solve/parameter entry points are explicitly unsupported. Second-order adjoints and finite-difference reference derivatives require CPU arrays. No CUDA/ROCm execution or performance result is claimed.
Typed retained history found during batch B verification
A full CPU regression run exposed an abstract inferred return type when retained
Broyden history crossed the native Enzyme custom rule. Reading the history from
Ref{Any} could leave a UnionAll for the report/preconditioner. Enzyme rejected
that ABI contract. Explicit rule-return annotations repaired reverse mode but
batched forward mode triggered an Enzyme LLVM assertion; those annotations were
removed. The solver now returns the same correction vectors as a typed third
result on its private retained-history path. Every exit uses the same result
helper. The public solve still returns (root, SolveReport), including the
existing optional history-holder keyword; residual evaluations, update arithmetic
and iteration counts are unchanged. No extra root or residual evaluation occurs.
The existing non-diagonal history oracles in test/lp002_batch_a.jl check reverse
and batched forward derivatives, both orientations, eviction and warm starts.
The focused reproduction also uses explicit bounds checking. This adds no
Enzyme device or nested-AD rule.
Finite-difference steps and residual norms (2026-10-29, LP-019 items 4 and 10)
fd_jacobian added to in the element type and divided by
the unrounded . In Float32 the probe rounds to , so the identity’s
Jacobian came out as 1.1920929; in Float16 the probe vanished and the Jacobian was 0. Even in
Float64 the identity gave . The quotient now uses the increment the rounded
probe actually makes (_fd_step, shared with flow.jl’s _divergence), and the default
relative step is , so Float64 keeps .
_norm was sqrt(sum(abs2, v)), which underflows: a constant residual [1e-200] with
tol = 1e-210 was reported converged at iteration 0 by Picard, Broyden and limited Broyden,
although there is no root. It is now LinearAlgebra.norm, which rescales. Tested in
test/lp019.jl.
After review (2026-10-30): the default step uses eps of the element type when it is defined,
which for a ForwardDiff dual is its primal’s (Float16 duals had kept and a zero
increment); a step that does not change throws. _norm returns the norm in the residual’s
own precision: converting to Float64 underflowed a BigFloat residual of .