comparison derivation

Zhang (2025) replaces self-attention by features of the 2-planes spanned by pairs of token states, written in Plücker coordinates. The algebra behind the layer is the Grassmannian and its Plücker embedding, whose coordinates are the same minors that express independence in algebraic statistics. This note records what the paper shows and what it does not, and what Lenticulum can take from it: a deterministic polynomial factor, a known variety to test algebraic fitting on, and as a parameter space (L-SPEC-004 Algebraic Factors Toric and Grassmannian).

Sources: Zhang, Attention Is Not What You Need, arXiv:2512.19428v1, 2025 (§3.2 for the layer, §6.4 for the limitations); Edelman, Arias & Smith, The Geometry of Algorithms with Orthogonality Constraints, SIAM J. Matrix Anal. Appl. 20(2), 1998; the facts about and the Plücker relations are standard multilinear algebra, derived below; the comparison with attention and the uses in Lenticulum are original to this vault; code: lib/AlgebraicFactors.jl/src/plucker.jl, lib/AlgebraicFactors.jl/src/polyfit.jl, veronese.jl, lib/AlgebraicFactors.jl/test/runtests.jl, test/polynomial.jl (forward map, moments, quadrics and real Klein nullspace fits).

Bibliography: Zhang 2025 · Edelman et al. 1998

Theory (CT-ML wiki): Parametric Lens

The layer (arXiv v1, §3.2)

Hidden states are reduced to (, in the paper). For each offset in a set (e.g. ) with , the pair gives

normalised to , mapped back by , averaged over the valid offsets, and gated into : with ; then LayerNorm and a feed-forward block. There is no softmax and no score matrix; the cost is linear in for fixed and .

The algebra (derivation)

Write , with coordinates for .

  1. Rank. iff and are linearly dependent, i.e. iff all minors of the matrix vanish: the determinantal condition that expresses independence in Algebraic Statistics Bridge. By Lagrange’s identity , the squared area of the parallelogram.
  2. Only the plane matters. . So depends on the pair only through the plane it spans and its oriented area. For , depends only on the oriented plane ( are its two orientations); the paper’s has this reading only when . This is the gauge of The Parameter is a Grassmannian, with .
  3. Linear maps. . A change of basis in the reduced space acts linearly on . Away from the floor, a conformal (, orthogonal) gives , which can absorb; a general adds a direction-dependent denominator.
  4. The image is a variety. The Plücker relations for cut out the affine cone of decomposable 2-vectors (zero included); removing zero and dividing by scalars gives , of dimension . For it is one quadric in (the Klein quadric). For the paper’s , the features lie on a cone of dimension in .
  5. Compared with attention (an interpretation). An attention score is one bilinear form in the pair. The Plücker features are antisymmetric bilinear forms , combined linearly afterwards. The pairing is fixed by the offsets, not chosen by a softmax over positions.

What the paper shows, and what it does not

  • Results. Wikitext-2 validation perplexity within about 10–15% of size-matched Transformers (13–18M parameters); on SNLI, a Grassmann head on DistilBERT at 0.8538 test accuracy against 0.8511 for a Transformer head. Single runs as reported, without variance across seeds; small scale; slower than attention at the lengths tried. No code is linked.
  • No theorem. Universal approximation is invoked informally; the claim that the features are more interpretable than attention is argued, not measured.
  • Causality, as written. §3.2 pairs with and calls this causal, “positions strictly to its right (future), consistent with left-to-right language modeling”. Read literally, the state at then depends on the token at , which next-token prediction at must not see, and the paper describes no shift or mask that prevents it. Without code this cannot be settled. A reproduction should pair with , and the Wikitext-2 numbers should not be relied on until one has.

What Lenticulum can take

  1. A Plücker factor. is a deterministic polynomial map of degree 2, bilinear, with a closed-form Jacobian. Implemented in AlgebraicFactors as plucker, plucker_jacobian (columns vcat(z,w)), and forward-only PluckerFactor(r). plucker_pairs specifies lexicographic coordinate order; plucker_relations and plucker_residuals supply the quadrics. Diracs and paired particles transport exactly; independent Gaussian inputs give exact moments and a moment-matched Gaussian message, subject to the explicit singular-covariance handling in plucker. The inverse polarity remains open: a nonzero bivector determines the pair only up to (a unit-normalised oriented one up to , a plane up to ), so the inverse message is a belief on a fibre, not a point; how to represent it is open, and needs separate treatment.
  2. A variety tested by fitting. Klein quadric fit_nullspace fits all 28 affine degree-two features in ℝ⁶ to 400 Gaussian-pair z∧w points, Xoshiro(5106). Both monomial and Chebyshev tables give exactly one null direction at rtol=1e-12; after basis conversion the fitted coefficients agree up to sign with plucker_relations(4) (normalised dot-product tolerance 2e-14). The monomial singular-value gap is 1.91054415134e14, Chebyshev 1.21729757389e14; polyfit records the full run. This settles §3(a)‘s fitting expectation in L-SPEC-004 Algebraic Factors Toric and Grassmannian. Unit-normalised points are expected to add ‖p‖²−1 as a second null direction; that case remains unverified by repository tests.
  3. as a parameter space, for the algebraic factor (The Parameter is a Grassmannian): a Stiefel representative, the projected gradient , a QR retraction (Edelman, Arias & Smith 1998). L-SPEC-004 §2.
  4. A causal Grassmann mixing block as an experiment for language models. It belongs in LenticulumLM, not in this library.

Related: The Parameter is a Grassmannian, Algebraic Statistics Bridge, Toric Models as Factors, Algebraic Implicit Learners