model derivation

Derivation. The full forward (jvp) and adjoint (vjp) equations for an algebraic factor. Cost: one linear solve. The parameter cotangent turns out to be rank at most one.

Sources: original derivation; code: lib/AlgebraicFactors.jl/src/polynomial.jl, lib/AlgebraicFactors.jl/test/polynomial.jl (exact message Jacobians and scalar least-squares mixed-Hessian check; IFT solve derivatives are not implemented).

Theory (CT-ML wiki): Lens

Implementation boundary

PolynomialFactor now uses exact r and J for single-branch Gaussian sites and iterated frozen-metric Gauss–Newton (polynomial). The jvp/vjp equations below remain a derivation: no IFT rule, saved root factorisation, solve differentiation or ExactCoupling declaration is implemented. Gauss–Newton posterior curvature must not be described as the exact Hessian at a nonzero residual.

Setup

, with clamped, inferred, , . Assume the well-posed case and invertible at the solution (Inference as Root Finding; failure is Branches and the Discriminant).

Write, using The Veronese Parametrisation §“Jacobians, exactly”,

All three are exact; no automatic differentiation is involved anywhere.

Total differential

Differentiating the constraint , which holds identically in a neighbourhood:

(the third term because is linear in , so its differential in is just ). Hence the implicit function theorem:

giving the two sensitivities

Forward mode (jvp)

Given tangents :

One solve. Cost for the factorisation, per additional tangent — so a factorisation of is reused across the whole batch of tangents.

Reverse mode (vjp) — the adjoint

This is what the put of a lens needs. Given a cotangent arriving from downstream, we want and satisfying for all tangents.

Substitute:

Define the adjoint variable

— one linear solve with the transpose. Then

using . Therefore

Three things to notice

1. The parameter cotangent has rank at most one

is an outer product. Every data point contributes a parameter cotangent of rank at most one (it can be zero), and their unprojected minibatch sum has rank at most B. The Grassmann projection/retraction is a separate operation.

Consequences:

  • The Grassmannian update of The Parameter is a Grassmannian is a rank- perturbation of a -plane — a subspace-tracking problem, costing rather than .
  • Storage of the accumulated gradient can be rather than .
  • This cotangent statement alone does not imply rank B for a Gauss–Newton matrix: that matrix uses the full residual parameter Jacobian.

This is a real structural gift and it comes directly from the linearity in .

2. The cost is one solve, shared

is factorised once during inference (Newton needs it anyway) and reused for the adjoint. So the backward pass is essentially free given the forward pass — one triangular solve. Compare an unrolled equilibrium solver, which pays in memory and time.

3. Where it breaks

At a root where Jᵤ is singular the square IFT does not apply. Inspect its smallest singular value as well as conditioning: a nonzero scalar has condition number one however close to zero it is (checked on the circle in the polynomial tests). See Branches and the Discriminant. Near the honest options are to regularise the solve (, i.e. a Tikhonov damping of the adjoint equation) and report the approximation, or refuse. This is the minimiser of ‖Jᵤᵀλ+x̄ᵤ‖²+ε‖λ‖²; the previous formula used the wrong transpose. Its normal equation is checked with a nonsymmetric 2×2 Jᵤ in Polynomial note corrections.

The overdetermined / minimising case

When there is generically no root and inference minimises instead. The stationarity condition for fixed M is . Differentiation gives

The mixed residual-Hessian term in Hᵤₒ was missing in the earlier formula. For r=(u²+ou−1,u−o), M=I, o=0.4, independently solved stationary points and central differences agree with this expression (atol=1e−9); omitting the mixed term gives an error above 1e−3. This is checked in Polynomial note corrections. If M varies with coordinates, differentiating its stationarity condition adds metric-derivative terms too.

Hᵤᵤ is the full Hessian of the fixed-metric objective. Dropping the residual second-derivative terms in both blocks gives the Gauss–Newton sensitivity −(JᵤᵀMJᵤ)⁻¹JᵤᵀMJₒ, agreeing at r=0 when the required inverse exists. This is a frozen-metric approximation when the Sampson metric varies. The second-derivative terms are also exactly computable here (the are again times a fixed monomial Hessian), so unlike in neural settings the Gauss–Newton approximation is a choice, not a necessity.

Why this is not “differential algebra”

The README’s table lists the implicit backward pass as “implicit function theorem / differential algebra”. Everything above is the implicit function theorem — ordinary multivariable calculus. No differential algebra appears, and none is needed. Differential algebra does have genuine roles here, but they are different jobs:

jobtoolnote
derivative of the solveimplicit function theoremthis note
is this polarity well-posed at all?elimination / dimension theoryComposition is Elimination
composing two factors into oneGröbner / border basesComposition is Elimination
factors that are DAEs, differential elimination, index reductionDifferential Algebra and DAE Factors

The last row is where differential algebra genuinely enters a backward pass, and even there it is index reduction of the adjoint DAE rather than differentiation as such.

Related: Inference as Root Finding, The Parameter is a Grassmannian, Lens, Differential Algebra and DAE Factors