For a smooth family of probability distributions on , parametrised by , the score is the gradient of the log-likelihood, , and the Fisher information matrix is its second moment (Nielsen, eqs. 79–80):
Read as a Riemannian metric on it is the Fisher information metric: the length of a parameter step is how far it moves the distribution, , not how far it moves the numbers . It is positive semidefinite, and definite for regular models.
Sources: Nielsen, An elementary introduction to information geometry arXiv:1808.08271 (notes) §3.9 (eqs. 79–80), §3.10 (Theorem 8); Martens arXiv:1412.1193 (notes) §5 (eqs. 3–4), §9.2; Karwowski, Nielsen & Staton arXiv:2610.17886 (notes) Definition 15, Propositions 8–9, Example 1.
Why this metric: invariance
Theorem 8 (Čencov; Nielsen §3.10). The Fisher information metric is the unique metric, up to a scaling constant, that is invariant under Markov embeddings.
A Markov embedding re-describes the sample space without adding or destroying information (refining outcomes into sub-outcomes in fixed proportions). Any statistically meaningful notion of distance between distributions must ignore such re-descriptions, and only the Fisher metric does. Under a change of parameters it transforms as a metric tensor, , which is exactly the condition under which the natural gradient does not depend on the parametrisation (Martens, Theorem 1).
Two examples
- A coin in log-odds: has score and Fisher information : large near a fair coin, vanishing as the coin becomes deterministic.
- A Gaussian with a parametrised mean: has , the Gauss–Newton matrix of least squares with metric . Martens (§9.2) shows that the Fisher equals the generalised Gauss–Newton matrix whenever the output distribution is an exponential family in its natural parameters: Gaussians with a parametrised mean, softmax categorical distributions. In that case second-order optimisation and information geometry are the same thing.
Composition: a chain rule for the Fisher metric
A probabilistic program is a composite of kernels, so one wants its Fisher metric from those of its parts. Karwowski, Nielsen & Staton do this for programs presented by their traces (Definition 15 is the Fisher metric of a program , taken over its random choices):
Proposition 8(7). , where pulls a metric on back to along the output map of , averaged over the trace (Definition 16).
The Fisher metric of a composite is the local Fisher metric of the first stage plus the downstream Fisher metric transported back, which is the same shape as the free-energy chain rule of a Statistical Game (Theorem 23: the upstream loss averaged, plus the downstream loss at the pushed-forward prior). The other properties are what one expects (Propositions 8–9): deterministic maps carry no Fisher information (); independent programs add (); sharing an input by copying adds the two metrics.
The price is extensionality. The Fisher metric of a program depends on how its randomness is written down, not only on the distribution it produces: a program with randomness it never uses (a coin flipped with bias and then ignored) has a nonzero Fisher metric although its output is a constant (Example 1). This is why the same paper’s category of traces is only a Copy-Discard Category, not a Markov Category. See Information Geometry of Probabilistic Programs for the full construction and how it compares with the metric of the output distribution.
In AutoBayes and Lenticulum
AutoBayes intends parameterized statistical games to be trained by descent with respect to the Fisher metric (Definition 27, Statistical Game). Lenticulum.jl keeps a vector energy so that this metric stays computable: for a Gaussian residual factor the Fisher metric in is , the Gauss–Newton matrix of the residual with the metric that the factor already uses to scalarise it (Scalar and Multivariate Energy §6 and §8).
Docs: Julia manual: mathematical operations — exact sums over finite traces.
σ(t) = 1 / (1 + exp(-t))
bern(p, b) = b == 1 ? p : 1 - p # mass of b ∈ {0, 1}
# A coin in log-odds: E[score²] = σ(1 − σ)
fisher_coin(α) = sum(bern(σ(α), b) * (b - σ(α))^2 for b in (0, 1))
fisher_coin(0.7) ≈ σ(0.7) * (1 - σ(0.7)) # true
# Chain rule (Karwowski et al., Prop. 8(7)) on F: x ⊢ b ← flip(x); return x + b
# and G: y ⊢ c ← flip(y); return c
x = 0.4
score_F(x, b) = b - σ(x) # ∂x log q(x, b)
score_G(y, c) = c - σ(y) # ∂y log r(y, c)
# the composite's trace is (b, c); its score is the sum of the two, by the chain rule of ∂x
g_GF = sum(bern(σ(x), b) * bern(σ(x + b), c) * (score_F(x, b) + score_G(x + b, c))^2
for b in (0, 1), c in (0, 1))
g_F = fisher_coin(x)
pullback_gG = sum(bern(σ(x), b) * fisher_coin(x + b) * 1^2 for b in (0, 1)) # F♯ g^G, velocity 1
g_GF ≈ g_F + pullback_gG # true: local + transported
# Example 1: a coin that is flipped and ignored. Output constant, Fisher metric not zero.
g_unused(θ) = sum(bern(σ(θ), b) * (b - σ(θ))^2 for b in (0, 1))
g_unused(0.0) > 0 # true: the metric is intensionalimport Mathlib
/-- The Fisher information of a one-parameter family `q θ` on a finite sample space,
`E_{ω ∼ q θ} [(∂_θ log q θ ω)²]` (Nielsen eq. 80, one-dimensional). -/
noncomputable def fisher {n : ℕ} (q : ℝ → Fin n → ℝ) (θ : ℝ) : ℝ :=
∑ ω, q θ ω * (deriv (fun t => Real.log (q t ω)) θ) ^ 2
/-- A family that does not depend on the parameter carries no Fisher information. -/
theorem fisher_const {n : ℕ} (p : Fin n → ℝ) (θ : ℝ) : fisher (fun _ => p) θ = 0 := by
simp [fisher]
/-- The Fisher information is a sum of nonnegative terms when the masses are nonnegative. -/
theorem fisher_nonneg {n : ℕ} (q : ℝ → Fin n → ℝ) (θ : ℝ) (hq : ∀ ω, 0 ≤ q θ ω) :
0 ≤ fisher q θ :=
Finset.sum_nonneg fun ω _ => mul_nonneg (hq ω) (sq_nonneg _)-- The Fisher metric of a coin, and its chain rule along a two-stage program
sigma :: Double -> Double
sigma t = 1 / (1 + exp (negate t))
bern :: Double -> Int -> Double
bern p b = if b == 1 then p else 1 - p
fisherCoin :: Double -> Double
fisherCoin a = sum [ bern (sigma a) b * (fromIntegral b - sigma a) ^ 2 | b <- [0, 1] ]
-- F: x ⊢ b ← flip(x); return x + b G: y ⊢ c ← flip(y); return c
composite :: Double -> Double
composite x = sum [ bern (sigma x) b * bern (sigma y) c
* ((fromIntegral b - sigma x) + (fromIntegral c - sigma y)) ^ 2
| b <- [0, 1], c <- [0, 1], let y = x + fromIntegral b ]
chainRule :: Double -> Double -- g^F + F♯ g^G
chainRule x = fisherCoin x + sum [ bern (sigma x) b * fisherCoin (x + fromIntegral b) | b <- [0, 1] ]
main :: IO ()
main = print (composite 0.4, chainRule 0.4) -- equal