Derivation. The parameter of an algebraic factor is not a matrix; it is a -plane. This is the cleanest concrete instance in the whole vault of Definition 2.5’s distinction between parameter points and updates. Grassmannian projection, QR retraction and wrapped GD/momentum are implemented; polynomial residuals, their Sampson scalarisation and fitting are implemented too.
Sources: original to this vault (design and analysis); Edelman, Arias & Smith (1998), The Geometry of Algorithms with Orthogonality Constraints; Absil, Mahony & Sepulchre (2008), Optimization Algorithms on Matrix Manifolds; code:
lib/AlgebraicFactors.jl/src/grassmann.jl,lib/AlgebraicFactors.jl/src/polynomial.jl,polyfit.jl,lib/AlgebraicFactors.jl/test/runtests.jl,lib/AlgebraicFactors.jl/test/polynomial.jl(real factor gauge invariance and full Sampson gradient checks).Bibliography: Edelman et al. 1998 · Absil et al. 2008
Theory (CT-ML wiki): Statistical Game · Parametric Lens · Lens · Gradient Based Learning with Parametric Lenses
The non-identifiability
If then , so
An invertible row change preserves the generated ideal and zero set. Conversely, different row spaces can still generate the same ideal or real locus (e.g. different generating sets, redundant polynomial multiples or real radicals). The fixed-degree residual subspace depends on Θ only through its row space; Grassmann parametrisation removes this row gauge, not every equivalence between variety descriptions. The parameter is therefore genuinely a point of the Grassmannian
not of (dimension ). The missing dimensions are pure gauge.
This is not pedantry. Three things go wrong if you ignore it:
- A residual loss can change along the gauge directions: changing the row scales or basis changes the residual while preserving its zero set. Euclidean distances between matrices therefore need not measure differences between relations. A truly gauge-invariant objective has no first-order derivative along gauge directions; drift is not inevitable for every objective.
- is a global minimiser of an unnormalised squared-residual objective, and it is reachable by descent. The normalisation of Fitting is a Nullspace Problem is not a regulariser bolted on to prevent collapse — it chooses a Stiefel representative of a point of , still modulo orthogonal row-basis changes. It is not a coordinate chart; collapse leaves the fixed-rank parameter space.
- Small raw residuals can come from small coefficients rather than a better geometric fit. Scaling a full-rank matrix towards zero preserves its Grassmann point at every nonzero scale; only the rank-deficient limit leaves the fixed-rank space. Orthonormal representatives remove that scale freedom.
The Riemannian gradient, derived
Represent a point by with (a Stiefel representative; the Grassmannian is the quotient by the action). The horizontal (tangent-to-) space at is
i.e. row-wise orthogonal to the current row space. Given a Euclidean gradient , the projection onto is
Check: . ✓
The update is then a retraction, not an addition:
where orthonormalises the rows (thin QR of the transpose, or a polar factor). Edelman–Arias–Smith is the standard reference for the geometry; the point here is only that .
This is exactly
Definition 2.5 carries a parameter pair with , and the vault’s note on it says the distinction matters “for a parameter constrained to a manifold, where the update lives in the tangent space, not in the manifold”. Here that is not a hypothetical:
and Definition 3.11’s gradient-update lens is type-incorrect. The correct lens is
which is still a perfectly good lens — the framework accommodates it without modification. The categorical setup was right and the naive implementation was wrong, which is a satisfying vindication of taking the distinction seriously.
The metric choice is not a Fisher-metric identification
Definition 27 uses the Fisher information metric of a statistical model. Here the canonical Grassmann metric on horizontal lifts is , and is the Riemannian gradient for that metric. No statistical model or Fisher information calculation in this note identifies the two metrics. The earlier claim that the projection automatically computes the Fisher natural gradient was unsupported. The implementation makes only the canonical-metric claim, checked by the first-order trace-objective descent test. The dense projection costs when evaluated as .
Interaction with the rank-one structure
Backpropagation by the Implicit Function Theorem derives that each data point contributes a rank-one cotangent . So a minibatch of size gives of rank , and
is also rank . The Riemannian update is a rank- perturbation of a -plane. If is retained in factored form, its projection can be formed by projecting before multiplying by , at cost . This does not make the complete QR update cost : the implemented dense retraction costs and does not exploit a supplied low-rank gradient representation. Forming an projector is unnecessary in either case.
Caveat: the Grassmannian is the right space only for a fixed and
Model selection — choosing how many generators and what degree — moves you between Grassmannians of different dimensions, and there is no smooth path. The numerical rank decision of Fitting is a Nullspace Problem is therefore a discrete jump between manifolds, not a continuous shrinkage within a fixed Grassmannian. polyfit implements fixed-rank algebraic and Sampson fitting, the latter using the canonical metric and the derivative of its varying covariance. It has no variable-rank relaxation; see Open Problems in Algebraic Implicit Learning §4.
Related: Grassmann Mixing and Plücker Coordinates, L-SPEC-004 Algebraic Factors Toric and Grassmannian, Fitting is a Nullspace Problem, Parametric Lens, Factors are Parameterized Statistical Games, Backpropagation by the Implicit Function Theorem