Derivation. The full forward (jvp) and adjoint (vjp) equations for an algebraic factor. Cost: one linear solve. The parameter cotangent turns out to be rank at most one.
Sources: original derivation; code:
lib/AlgebraicFactors.jl/src/polynomial.jl,lib/AlgebraicFactors.jl/test/polynomial.jl(exact message Jacobians and scalar least-squares mixed-Hessian check; IFT solve derivatives are not implemented).Theory (CT-ML wiki): Lens
Implementation boundary
PolynomialFactor now uses exact r and J for single-branch Gaussian sites and
iterated frozen-metric Gauss–Newton (polynomial). The jvp/vjp equations below
remain a derivation: no IFT rule, saved root factorisation, solve differentiation
or ExactCoupling declaration is implemented. Gauss–Newton posterior curvature
must not be described as the exact Hessian at a nonzero residual.
Setup
, with clamped, inferred, , . Assume the well-posed case and invertible at the solution (Inference as Root Finding; failure is Branches and the Discriminant).
Write, using The Veronese Parametrisation §“Jacobians, exactly”,
All three are exact; no automatic differentiation is involved anywhere.
Total differential
Differentiating the constraint , which holds identically in a neighbourhood:
(the third term because is linear in , so its differential in is just ). Hence the implicit function theorem:
giving the two sensitivities
Forward mode (jvp)
Given tangents :
One solve. Cost for the factorisation, per additional tangent — so a factorisation of is reused across the whole batch of tangents.
Reverse mode (vjp) — the adjoint
This is what the put of a lens needs. Given a cotangent
arriving from downstream, we want and
satisfying
for all tangents.
Substitute:
Define the adjoint variable
— one linear solve with the transpose. Then
using . Therefore
Three things to notice
1. The parameter cotangent has rank at most one
is an outer product. Every data point contributes a parameter cotangent of rank at most one (it can be zero), and their unprojected minibatch sum has rank at most B. The Grassmann projection/retraction is a separate operation.
Consequences:
- The Grassmannian update of The Parameter is a Grassmannian is a rank- perturbation of a -plane — a subspace-tracking problem, costing rather than .
- Storage of the accumulated gradient can be rather than .
- This cotangent statement alone does not imply rank B for a Gauss–Newton matrix: that matrix uses the full residual parameter Jacobian.
This is a real structural gift and it comes directly from the linearity in .
2. The cost is one solve, shared
is factorised once during inference (Newton needs it anyway) and reused for the adjoint. So the backward pass is essentially free given the forward pass — one triangular solve. Compare an unrolled equilibrium solver, which pays in memory and time.
3. Where it breaks
At a root where Jᵤ is singular the square IFT does not apply. Inspect its smallest
singular value as well as conditioning: a nonzero scalar has condition number
one however close to zero it is (checked on the circle in the polynomial tests).
See Branches and the Discriminant.
Near the honest options are to regularise the solve
(, i.e. a
Tikhonov damping of the adjoint equation) and report the approximation, or refuse.
This is the minimiser of ‖Jᵤᵀλ+x̄ᵤ‖²+ε‖λ‖²; the previous formula used the
wrong transpose. Its normal equation is checked with a nonsymmetric 2×2 Jᵤ in
Polynomial note corrections.
The overdetermined / minimising case
When there is generically no root and inference minimises instead. The stationarity condition for fixed M is . Differentiation gives
The mixed residual-Hessian term in Hᵤₒ was missing in the earlier formula.
For r=(u²+ou−1,u−o), M=I, o=0.4, independently solved stationary points
and central differences agree with this expression (atol=1e−9); omitting the
mixed term gives an error above 1e−3. This is checked in Polynomial note corrections.
If M varies with coordinates, differentiating its stationarity condition adds
metric-derivative terms too.
Hᵤᵤ is the full Hessian of the fixed-metric objective. Dropping the residual second-derivative terms in both blocks gives the Gauss–Newton sensitivity −(JᵤᵀMJᵤ)⁻¹JᵤᵀMJₒ, agreeing at r=0 when the required inverse exists. This is a frozen-metric approximation when the Sampson metric varies. The second-derivative terms are also exactly computable here (the are again times a fixed monomial Hessian), so unlike in neural settings the Gauss–Newton approximation is a choice, not a necessity.
Why this is not “differential algebra”
The README’s table lists the implicit backward pass as “implicit function theorem / differential algebra”. Everything above is the implicit function theorem — ordinary multivariable calculus. No differential algebra appears, and none is needed. Differential algebra does have genuine roles here, but they are different jobs:
| job | tool | note |
|---|---|---|
| derivative of the solve | implicit function theorem | this note |
| is this polarity well-posed at all? | elimination / dimension theory | Composition is Elimination |
| composing two factors into one | Gröbner / border bases | Composition is Elimination |
| factors that are DAEs, | differential elimination, index reduction | Differential Algebra and DAE Factors |
The last row is where differential algebra genuinely enters a backward pass, and even there it is index reduction of the adjoint DAE rather than differentiation as such.
Related: Inference as Root Finding, The Parameter is a Grassmannian, Lens, Differential Algebra and DAE Factors