The natural gradient of a loss on a statistical model is the ordinary gradient with its index raised by the Fisher Information Metric (Amari 1998; Nielsen eq. 177):
The ordinary gradient is a covector: it says how changes per unit change of each coordinate, and its meaning changes when the coordinates do. A metric turns it into a vector, the direction of steepest descent per unit of distance; with the Fisher metric, distance is measured between the distributions themselves. Natural gradient descent is the first-order approximation of Riemannian gradient descent on the statistical manifold, using the retraction instead of the exponential map (Nielsen §4.1.2, eqs. 172–178).
Sources: Amari, Natural gradient works efficiently in learning, Neural Computation 10(2):251–276 (1998), doi:10.1162/089976698300017746; Nielsen arXiv:1808.08271 (notes) §4.1.2 (eqs. 172–178, Property 5); Martens arXiv:1412.1193 (notes) §5, §9.2, §12 (Theorem 1), §14.1; Karwowski, Nielsen & Staton arXiv:2610.17886 (notes) §2.6, §8.5 (Theorem 12, Definition 56).
Why: invariance under reparametrisation
Property 5 (Nielsen). The natural gradient is invariant under coordinate transformations.
Theorem 1 (Martens §12). If and the curvature matrices satisfy , then the update and the update agree to first order: , exactly when is affine.
The Fisher metric satisfies the condition of Theorem 1 because it is a metric tensor. So training a coin in log-odds or in probabilities, a Gaussian in or in , gives the same trajectory to first order. Plain gradient descent does not have this property: its trajectory depends on the chosen coordinates (see the code below). Amari also showed that natural gradient with a step size is asymptotically Fisher-efficient, attaining the Cramér–Rao bound (Martens §14.1).
Natural gradient in practice
- It is a second-order method. For a Gaussian or softmax output, the Fisher equals the generalised Gauss–Newton matrix (Martens §9.2). Natural gradient is then Gauss–Newton. K-FAC and Adam’s diagonal approximate this matrix (Karwowski et al. §2.6), and the link Ollivier found between natural gradient and the extended Kalman filter rests on the same condition (Martens §9.2).
- The cost is the metric. is and dense for a dense model. Programs and factor graphs are not dense: their metric decomposes along their structure, which is the motivation of Karwowski, Nielsen & Staton (§2.6). Their chain rule (Fisher Information Metric) builds the metric of a program from those of its parts.
- Degenerate metrics. A program’s metric can be only semidefinite: a parameter that does not influence the output, or influences it only through randomness that is discarded. Karwowski et al. then define the natural gradient in the quotient by the kernel of the metric (Definition 56). It exists iff vanishes on that kernel, and in coordinates it is computed with the Moore–Penrose pseudo-inverse, . For observations fed to copies of one program, the metrics of the copies add (Theorem 12).
In AutoBayes and Lenticulum
The intended semantics of a parameterized Statistical Game is descent with respect to the Fisher metric (AutoBayes Definition 27), the “Bayesian learning rule”. The local gradient composition of Definition 29 drops terms (Remark 30). A natural gradient built from the chain rule of the Fisher metric is one way to say compositionally what the full curvature is. Lenticulum.jl’s vector energies keep the Jacobians this needs (Scalar and Multivariate Energy §6).
Docs: LinearAlgebra: pinv — the Moore–Penrose representative of Definition 56.
using LinearAlgebra
σ(t) = 1 / (1 + exp(-t))
p̂ = 0.8 # data: 80% heads; ℓ = average negative log-likelihood
# in log-odds α: ∇ℓ = σ(α) − p̂, I(α) = σ(α)(1 − σ(α))
# in probability μ: ∇ℓ = (μ − p̂)/(μ(1−μ)), I(μ) = 1/(μ(1−μ))
α = -0.5; μ = σ(α); η = 0.01
nat_α = (σ(α) - p̂) / (σ(α) * (1 - σ(α)))
nat_μ = ((μ - p̂) / (μ * (1 - μ))) * (μ * (1 - μ))
# pushed through dμ/dα = σ(1 − σ), the log-odds step is the probability step: invariance
isapprox(σ(α) * (1 - σ(α)) * nat_α, nat_μ) # true
# the plain gradient is not invariant: the two coordinates disagree on the step
plain_α_in_μ = σ(α) * (1 - σ(α)) * (σ(α) - p̂)
plain_μ = (μ - p̂) / (μ * (1 - μ))
isapprox(plain_α_in_μ, plain_μ) # false
# Definition 56: a degenerate metric. (α, β) ⊢ flip(α), where β is never used.
G = [σ(α) * (1 - σ(α)) 0.0; 0.0 0.0] # β-direction is in the kernel
∇ℓ = [σ(α) - p̂, 0.0] # dℓ vanishes on the kernel, so the gradient exists
pinv(G) * ∇ℓ ≈ [nat_α, 0.0] # true: no step in the unused directionimport Mathlib
open Matrix
/-- The natural gradient `I⁻¹ ∇ℓ` for a Fisher matrix `I` and a gradient `g`. -/
noncomputable def natGrad {n : ℕ} (I : Matrix (Fin n) (Fin n) ℝ) (g : Fin n → ℝ) : Fin n → ℝ :=
I⁻¹ *ᵥ g
/-- Raising the index and lowering it again gives the gradient back, when `I` is invertible. -/
theorem mulVec_natGrad {n : ℕ} (I : Matrix (Fin n) (Fin n) ℝ) (g : Fin n → ℝ)
(h : IsUnit I.det) : I *ᵥ natGrad I g = g := by
simp [natGrad, mulVec_mulVec, mul_nonsing_inv _ h]
#check @Matrix.PosDef -- a regular model's Fisher matrix is positive definite-- Natural versus plain gradient for a coin, in two coordinate systems
sigma :: Double -> Double
sigma t = 1 / (1 + exp (negate t))
main :: IO ()
main = do
let pHat = 0.8
a = -0.5
mu = sigma a
jac = sigma a * (1 - sigma a) -- dμ/dα
natA = (sigma a - pHat) / jac -- I(α)⁻¹ ∇_α ℓ
natM = (mu - pHat) -- I(μ)⁻¹ ∇_μ ℓ
plainA = sigma a - pHat
plainM = (mu - pHat) / (mu * (1 - mu))
print (jac * natA, natM) -- equal: natural gradient is invariant
print (jac * plainA, plainM) -- different: plain gradient is not