New Insights and Perspectives on the Natural Gradient Method — James Martens (2014). arXiv:1412.1193 (v11, PDF); Journal of Machine Learning Research 21(146) (2020) 1–76.
A critical analysis of natural gradient descent: it is a second-order method whose Fisher matrix coincides with the generalised Gauss–Newton matrix for exponential-family outputs, its parametrisation invariance holds only to first order, the empirical Fisher is a poor substitute, and its convergence and efficiency are analysed asymptotically.
Sources: the paper, arXiv:1412.1193v11, checked against the arXiv listing and the JMLR page (21(146):1–76). Index: Papers.
Key definitions and results
- §5, eqs. 3–4: the Fisher matrix and the natural gradient
- §8: the generalised Gauss–Newton matrix
- §9.2: Fisher equals the GGN when the output distribution is an exponential family in its natural parameters
- §11: the empirical Fisher and why it differs
- §12, Theorem 1, Corollary 2: parametrisation invariance to first order, and in the limit of small steps
- §14.1: Amari’s Fisher efficiency result; §14.2, Theorems 3, 5: asymptotic convergence speed