implementation

NeuralODEFactor exposes both endpoints of a numerical flow. Reverse integration approximates the continuous inverse; discrete exactness depends on the integrator.

Sources: Hairer, Lubich, Wanner, Geometric Numerical Integration, 2nd ed., II.3 (adjoint and symmetric methods); code: neuralode.jl, flow.jl, ../examples/flow_roundtrip.jl, ../test/roundtrip.jl, ../test/sciml.jl, ../test/parameter_ad.jl, ../ext/ImplicitLayersEnzymeCoreActivityExt.jl

Bibliography: Hairer et al. 2006

Theory (CT-ML wiki): Bayesian Inversion · Bayesian Lens · Variational Free Energy · Reverse Derivative Category · Lens

Graph free-energy edge directions

NeuralODEFactor declares Mycelium.uses_cavity == true: its local free energy evaluates the endpoint residual from incoming points, returning zero if an endpoint is absent. Graph free-energy evaluation rejects an attached Emitting() edge with ArgumentError; use Bidirectional() if that channel must also send messages. Missing messages on permitted edges retain their existing local behaviour. See free_energy and LP-014 Emitting edges drop the cavity message from the free energy.

1. Two directions, both integrations

Both are the same integrator with the endpoints swapped (flow §1). Compare the other bidirectional factors in the project:

factordirection Adirection Bcost of B
GaussianFactorpushforwardlikelihooddifferent arithmetic, both closed-form
LinearConstraintFactorsolve for solve for identical; acausal by construction
DEQFactorroot-find for root-find for may not converge; needs a square residual
NeuralODEFactorintegrateintegratesame scheme; work and accuracy depend on direction

supported_polarities always returns two elements, with no dimension condition. This is an interface capability, not an accuracy or convergence guarantee.

2. The inversion label describes the discrete map

The residual uses the numerical forward map, . The exact continuous flow is invertible wherever the ODE has a unique solution over the interval in both directions; this does not make a discretisation reversible.

assemble uses inversion_kind(integrator, t₀, t₁). Euler, RK4 and adaptive SciML solves receive SolverInversion(integrator), the existing nonexact solver label. No shared core types change. Both inversion labels use the same endpoint integration and point lens game.

Fixed-step ImplicitMidpoint and Trapezoid are symmetric: reversing a step inverts it (Hairer–Lubich–Wanner, II.3). ExactInversion is used only with adaptive=false, a positive dt dividing the interval, no custom step limiter, and only the whitelisted solve options dt, adaptive, abstol, reltol, maxiters. The matching grid matters for time-dependent fields: a shortened final step would occur at the other end on reversal. Exactness here is up to nonlinear-solve tolerance and roundoff, not continuous-flow accuracy, and presumes converged stages and deterministic dynamics. Nonempty Lux state conservatively forces SolverInversion, since state updates may change the field. See flow for the extension.

The lens game scores the actual endpoint residual. Forward-produced endpoints satisfy their own residual even with a poor integrator; reverse-produced endpoints need not.

3. A flow cannot change dimension

NeuralODEFactor takes one dim, not two, because lives in one space for all . That is why channels gives both channels the same space, and it is a real modelling restriction: a NeuralODE cannot be an encoder that compresses. DEQFactor’s two channels are independent; this one’s are the same space at two times.

It is also why the density story of flow §3 works at all — a change of variables needs a bijection between spaces of equal dimension.

4. Implementation difficulties

4.1 State threading depends on the integration path

Built-in Euler/RK4 endpoint flows use a functional fold carrying (z, st) from step to step. RK4 threads state through its four stages in order. There is no Ref mutation on this path, so Zygote can differentiate the actual discrete map. The public vectorfield, adaptive SciML and divergence paths retain their existing Ref interface.

For a stateless dynamics layer that is correct. For one with running statistics it is arbitrary — and worse here than in deq §4.1, because RK4 evaluates the field four times per step at intermediate points that are not on the trajectory. A BatchNorm inside a NeuralODE would accumulate statistics from points the solution never visits. Adaptive solves also thread state through rejected trials; implicit solves and divergence probes add more evaluations. This preserves existing threading, but state-dependent dynamics need not define a deterministic ODE or a reversible map.

4.2 flow_logdet exists and nothing can use it

It computes the divergence correction for transporting a density, but inversion still returns a DiracBelief. GaussianBelief is available in LenticulumCore; a general non-Gaussian pushforward belief is not implemented, so the correction does not yet transport beliefs through this factor.

4.3 Truncation error and inverse conditioning are separate

Reversing Euler or RK4 does not invert their forward step even in exact arithmetic. For , , Euler’s forward multiplier is , its reverse multiplier is , whereas the discrete inverse is . Thus the relative endpoint round-trip residual is

Euler has an round trip. RK4 has forward global error and an round-trip bound for smooth dynamics on a fixed interval with bounded error amplification. For this scalar linear round trip the sharp order is fifth, not fourth. Its polynomial satisfies

Consequently doubling gives ratios approaching 2 (Euler round trip), 32 (linear RK4 round trip), and 16 (RK4 forward error against ). test/roundtrip.jl asserts all three: Euler at within 4%, RK4 at within 2% for the round trip and 4% for the forward error. These grids keep small; the margins allow finite-step corrections. BigFloat at 256 bits avoids the Float64 cancellation floor.

Reproduce this table with:

julia --project=lib/ImplicitLayers.jl lib/ImplicitLayers.jl/examples/flow_roundtrip.jl

The script starts from endpoint 3 on , reverses, then measures and . Values below are its 256-bit results, rounded to six digits; tests also check their exact discrete polynomial formulas.

schemeaNrelative residualrelative start error
Euler0.512.500000e-011.756394e-01
Euler0.5259.952147e-035.054864e-03
Euler0.5504.987770e-032.513628e-03
Euler0.51002.496909e-031.253396e-03
Euler0.52001.249223e-036.258477e-04
Euler-5.012.500000e+019.595723e-01
Euler-5.0256.396033e-013.572253e-01
Euler-5.0503.949939e-012.090267e-01
Euler-5.01002.214430e-011.139515e-01
Euler-5.02001.175376e-015.962588e-02
RK40.512.237956e-043.959794e-04
RK40.5252.222333e-116.778736e-10
RK40.5506.944531e-134.201538e-11
RK40.51002.170146e-142.615041e-12
RK40.52006.781689e-161.630999e-13
RK4-5.018.951823e+025.595067e-01
RK4-5.0252.233357e-055.645323e-05
RK4-5.0506.953127e-073.833894e-06
RK4-5.01002.170817e-082.497951e-07
RK4-5.02006.782214e-101.594056e-08

At the default , Euler with has residual 0.395 and start error 20.9%. RK4 gives there, but residual 895.18 at . Neither label nor fixed steps make these discrepancies harmless.

Independently of discretisation, inverting a contracting continuous flow amplifies absolute endpoint perturbations by (the norm of the inverse derivative for scalar ). For this is about 148.4. This is absolute error amplification, not a scalar relative condition number of 148.4. A symmetric integrator can invert its own discrete map without accurately recovering the continuous start point or removing this conditioning. test/sciml.jl verifies symmetric round trips at abstol=reltol=1e-12, and measures Tsit5 residuals as its tolerance is tightened; neither promises a global residual bound.

4.4 local_free_energy is identically zero on the relation

, and if came from then by construction. So the free energy is a good invariant (the tests use it as one) and a useless diagnostic: it cannot distinguish a well-integrated flow from a badly-integrated one, because both satisfy their own residual.

Reverse-solved endpoints can have nonzero forward residual from round-trip error. The lens game retains that residual. Comparing with a finer integration would estimate discretisation error; the game does not perform that comparison.

4.5 The prior argument is genuinely unused

Unlike deq, where it warm-starts the solver, a flow has nothing to warm-start. The inversion accepts and ignores it. The lens game separately scores its anchor cost at the returned point; it does not change the flow to condition on the prior.

Related: flow, deq, luxfactor, NeuralODE as an Invertible Factor, The Equilibrium Family, Inversions and Bayesian Lenses

Incoming factor game and solved-point lens game (LP-009)

free_energy(f, π, y, ps, st) follows GaussianFactor: π is the incoming Dirac on the first (start) channel, y the incoming Dirac on the second (end) channel. This fixed order corresponds to start unobserved, end observed; it does not depend on an inversion polarity. It delegates to local_free_energy, with scalar zero entropy. Non-Dirac arguments throw.

lens = first(assemble(f, p, ps, st)) restricts the flow relation to polarity p. free_energy(lens, π, inputs, ps, st) evaluates the endpoint residual at invert’s output plus an anchor cost on the unobserved endpoint. inputs is the named Dirac observation or that Dirac directly. This is Definition 20’s expectation under the point inversion. Entropy is scalar zero. Trivial priors cost zero; a Dirac prior costs infinity when the solved point differs, zero when equal; a proper Gaussian costs ½(u*-μ)'Λ(u*-μ), relative to its mean, without normalization constants. Channel precision does not soften Diracs. Soft observations, singular Gaussian anchors and other belief families throw. The inversion itself still ignores the prior, so this is scoring its output, not a prior-conditioned posterior flow.

For dz/dt=0, endpoints 1,3 give incoming energy ½(3-1)²=2. Solving either endpoint gives residual zero. The unknown-endpoint Jacobian is +1 or -1, so a Gaussian anchor centered on the incoming unknown, with precision J'J=1, costs ½(3-1)²=2 at the solution. test/conformance.jl checks this identity in both polarities, full factor conformance without game_gap, lens energies and incompatible hard priors. The same identity is exact for any residual affine in the unknown, with precision J'J; a nonlinear residual gives the first-order Gauss–Newton approximation described and tested in deq. It presumes a zero residual at the solved point; finite integration error need not satisfy that premise.

Scoring arbitrary endpoints uses energy/scalar_energy (the pointwise level of L-ADR-0003 Pointwise energy and free energy are separate levels), not a keyword on free_energy. Shared implementation and limitations: point_game.

Parameter derivatives (LP-002, 2026-10-26)

The optional core DI extension differentiates built-in Euler/RK4 endpoint energies with user-selected backends. This is discretise-then-differentiate, including functional Lux state updates, not a continuous adjoint. test/parameter_ad.jl checks ForwardDiff, Zygote, Mooncake, runtime-activity Enzyme (Reverse and Forward) and FD against scalar linear dynamics: with and multiplier ,

Euler has ; RK4 has . Tests also compare AD and FD for state-dependent stage dynamics, and fit a known Euler parameter through the graph’s summed local free energy using each AD backend.

Loading SciMLSensitivity activates an endpoint extension for SciMLIntegrator; see flow for its sensitivity algorithm and state policy. The parameter vector is an explicit ODEProblem.p, rebuilt using the core’s tree helpers, so adjoints see every Lux parameter. Recursively empty state is supported with the default :threaded policy. Nonempty-state primal fallback and its AD rejection remain unchanged. state_policy=:fixed extends sensitivities to functional nonempty state, held constant during the solve and advanced once at the final endpoint. Built-in Euler/RK4 support the same opt-in policy; their default still threads state through every stage. This distinction corrects an ambiguity in the batch A request: advance-once was the DEQ contract, not the built-in flow contract.

flow records the solver/backend matrix, state semantics and limitations. test/lp002_batch_a.jl uses state that affects the field and changes on update, so a stage-threaded implementation cannot accidentally pass. It compares both time directions, endpoint input/parameter derivatives, state updates and tight RK4 reference flows. Fixed-state flow_logdet is explicitly unsupported. Runtime-activity Enzyme reverse is covered for default Tsit5 sensitivities; Enzyme forward fails in upstream callback initialization. GPU and second-order flow sensitivities remain untested.

Enzyme needs runtime activity for built-in endpoint flows (SciML supports only the tested reverse configuration):

using Enzyme, ADTypes, DifferentiationInterface
ad = AutoEnzyme(; mode = Enzyme.set_runtime_activity(Enzyme.Reverse))
# Alternatively: AutoEnzyme(; mode = Enzyme.set_runtime_activity(Enzyme.Forward))
J = energy_jacobian(f, z0, nothing, y, ps, st; wrt = :ps, ad)
g = energy_gradient(f, z0, nothing, y, ps, st; ad)

The functional stage fold carries constant model/stage memory through an active computation. Static activity analysis cannot always distinguish that memory from active storage; without runtime activity the flow can raise EnzymeRuntimeActivityError. This is an activity configuration requirement, not an unsupported flow derivative. Enzyme’s runtime-activity documentation explains the memory checks enabled by this mode.

The optional EnzymeCore activity extension checks ad.mode with EnzymeCore.runtime_activity (verified with EnzymeCore 0.8.26). Core’s factor derivative entry points and Mycelium’s graph gradient call _check_factor_ad before entering AD. For built-in Euler/RK4, a default or explicitly non-runtime mode raises an ArgumentError containing both configurations above. The check does not need ForwardDiff; the separate DEQ IFT extension still does. There are no custom _flow Enzyme rules: normal differentiation handles the stages. An arbitrary outer DI function bypasses these factor entry checks and must select runtime activity itself.

test/parameter_ad.jl checks both Enzyme modes against the discrete closed forms for Euler/RK4 Jacobians, scalar energy and free-energy gradients, against finite differences for ADCountingDynamics with stage-dependent state, and in graph parameter_step learning with Nesterov look-ahead. It also verifies the configuration error. DEQ IFT and scored LuxFactor paths retain their passing plain AutoEnzyme() checks; they do not require this runtime-activity setting.

Device coverage and input-gradient repair (2026-10-27)

The built-in threaded flow used foldl(...; init=(u,st)). On the tested Zygote stack, the input in that initializer received no cotangent, although parameter closure gradients worked. A ChainRules reverse helper now performs the identical sequence of functional _lux_step calls in a plain loop and preserves the input derivative. The original fold remains the primal used by native backends. The test compares CPU primal results bit for bit with the former fold and checks input derivatives against the discrete linear map. Neither stage order nor state threading changes.

For device states, the GPUArraysCore extension converts times and step coefficients to state precision, including the final fixed-policy state evaluation. A time-dependent fixture asserts the incoming time precision at every evaluation and compares with the Euler/RK4 stability polynomial of an augmented constant linear system (, with an extra constant coordinate). JLArrays tests cover forward/reverse flows, Zygote and ForwardDiff input/parameter gradients, energy/free-energy gradients and the graph parameter driver, in Float32 and Float64. Functional nonempty-state stage threading and fixed-state endpoint updates are also checked against CPU references. SciML device solvers and divergence remain outside this coverage. Enzyme/Mooncake device parameter AD is rejected; their CPU flow support is retained. See flow and solve.

An initial full run exposed an Enzyme forward runtime-activity BLAS error when the loop was used as the general primal. Keeping the existing fold primal and using the loop only in ChainRules reverse preserves the previously tested native Enzyme/Mooncake paths; the existing 80-assertion flow testset is rerun unchanged.