Markov Categories and Entropy — Paolo Perrone (2022). arXiv:2212.11719 (v2, PDF); IEEE Transactions of Information Theory 70(3), 2024.
Combines Markov categories with divergences and metrics on hom-sets, defining mutual information as distance from independence and entropy as distance from determinism, recovering Shannon and Rényi entropies and data-processing inequalities.
Sources: the paper, arXiv:2212.11719v2, checked against the arXiv listing. Index: Papers.
Key definitions and results
- Definition 2.1, Remark 2.2: divergences, and how they relate to Lawvere metrics
- Definition 2.5, Theorem 2.7: divergences on a Markov category; equivalent data processing conditions
- Definition 2.9, Propositions 2.10, 2.13, 2.18, 2.22: KL, Rényi and total variation divergences on FinStoch and Stoch
- Definition 3.1, Proposition 3.2: mutual information as divergence from independence; data processing inequality
- Definition 4.1, Proposition 4.2: entropy as divergence from determinism; data processing for deterministic maps
- Theorem 4.6, Corollary 4.7: the KL entropy is Shannon entropy for discrete measures and infinite otherwise
- Theorem 4.10: the total variation entropy is the probability that two independent draws differ
Concept notes
Entropy in a Markov Category, Markov Category, Conditional Independence