Para Construction

definition example theorem

For a (strict) symmetric Monoidal Category $(\mathcal{C},\otimes ,I)$ — or more generally an Actegory — the parametrised category $\mathbf{Para}(\mathcal{C})$ has

• objects: the objects of $\mathcal{C}$;
• morphisms $A\to B$: pairs $(P,f)$ of a parameter object $P$ and a map $f:P\otimes A\to B$;
• identity on $A$: $(I,{\lambda }_A:I\otimes A\to A)$;
• composition of $(P,f):A\to B$ and $(Q,g):B\to C$: the pair

$$(Q\otimes P,(1_Q\otimes f);g):A\to C.$$

Parameter spaces multiply under composition: composing a layer with 20 weights and a layer with 30 weights gives one arrow with 50 weights, and the composite’s parameter wire is the pair of the two. A reparametrisation of $(P,f)$ along $\alpha :Q\to P$ is $(Q,(\alpha \otimes 1_A);f)$ — “feed the parameter wire through $\alpha$ first”. Reparametrisations are the 2-cells that make $\mathbf{Para}(\mathcal{C})$ a Bicategory; quotienting them out (Cruttwell et al.) gives a category.

Sources: Cruttwell, Gavranović, Ghani, Wilson & Zanasi, Categorical Foundations of Gradient-Based Learning arXiv:2103.01931 (notes) Definitions 2.1, 2.3, Example 2.2, Remark 2.1; Capucci, Gavranović, Hedges & Fjeldgren Rischel arXiv:2105.06332 (notes) Definition 2, Remarks 3–4, Proposition 5; Fong, Spivak & Tuyéras, Backprop as Functor arXiv:1711.10455 (notes) (the category $\mathbf{Para}$ of parametrised functions); Gavranović, Fundamental Components of Deep Learning arXiv:2403.13001 (notes) ch. 3.

Example: neural networks (Cruttwell et al., Example 2.2)

Let $\mathbf{Smooth}$ have natural numbers as objects and smooth maps ${\mathbb{R}}^n\to {\mathbb{R}}^m$ as morphisms, with $\otimes =+$ on objects. Then

$$\text{Dense(3 => 5, tanh)}=({\mathbb{R}}^{20},(W,b,x)\mapsto \tanh (Wx+b)):3\to 5$$

is a morphism of $\mathbf{Para}(\mathbf{Smooth})$ (15 weights plus 5 biases), and a two-layer chain $3\to 5\to 2$ has parameter object ${\mathbb{R}}^{12}\otimes {\mathbb{R}}^{20}$ — the second layer’s parameters first, exactly as in the composition rule. Deep-learning libraries store this as a nested record, e.g. Lux.jl’s (layer_1 = …, layer_2 = …), which is an order-insensitive representative of the associativity class.

Reparametrisations are everywhere

reparametrisation $\alpha :Q\to P$	what it does
$\Delta :P\to P\otimes P$	weight tying (two layers share one parameter)
a hypernetwork $Q\to P$	parameters generated by another network
$(W_0,A,B)\mapsto W_0+AB$	LoRA: a low-rank reparametrisation of a frozen weight
a quantisation map	low-precision weights
an optimiser’s get	optimisers — see Gradient-Based Learning with Parametric Lenses

Para is a monad, and CoPara is its dual

Capucci et al. (Proposition 5) show $\mathbf{Para}(-)$ is a monad on $\mathcal{M}$-actegories: a map parametrised twice, by $M$ and then by $N$, is a map parametrised once by $M\otimes N$. The dual $\mathbf{CoPara}(\mathcal{C})$ (Remark 4) has morphisms $X\to M\bullet Y$ — the extra object sits in the codomain. Probabilistic models with latent variables are coparametrised: an open model $X⇝[[p]]\times Y$ in the AutoBayes framework is a coparametrised Markov kernel whose “coparameter” is the latent space, composed by copy-composition (St Clere Smithe’s ${\mathbf{Copara}}_2$, thesis arXiv:2212.12538 (notes) §5.2.1).

Where it goes next

• $\mathbf{Para}(\mathbf{Lens}(\mathcal{C}))$ is the category of parametric lenses — the shape of a trainable layer with a backward pass.
• $\mathbf{Para}(R):\mathbf{Para}(\mathcal{C})\to \mathbf{Para}(\mathbf{Lens}(\mathcal{C}))$, for a reverse derivative $R$, is automatic differentiation.
• Parameterized statistical games (AutoBayes Definition 27) are $\mathbf{Para}$ applied to a category of Bayesian lenses with losses.
• Open games are $\mathbf{Para}(\mathbf{Optic}(\mathcal{C}))$ equipped with a selection (best-response) functor.

JuliaLeanHaskell

Docs: plain Julia — Catlab has no dedicated API for this; related: Catlab v0.16 docs · GATlab standard library

# Para(Smooth) on dense layers: a morphism is (parameter shape, f(p, x)).
struct ParaMap{F}; nparams::Int; f::F; end
(φ::ParaMap)(p, x) = φ.f(p, x)
function dense(n, m, σ = identity)
    ParaMap(m * n + m, (p, x) -> σ.(reshape(p[1:m*n], m, n) * x .+ p[m*n+1:end]))
end
# composition: parameters concatenate (second layer first, as in Q ⊗ P)
compose(g::ParaMap, f::ParaMap) =
    ParaMap(g.nparams + f.nparams, (p, x) -> g(p[1:g.nparams], f(p[g.nparams+1:end], x)))
net = compose(dense(5, 2), dense(3, 5, tanh))
net.nparams                                            # 12 + 20 = 32
p = randn(net.nparams); x = randn(3)
size(net(p, x))                                        # (2,)
# reparametrisation = weight tying: one 2-vector drives both halves of a 4-parameter map
twoscales = ParaMap(2, (p, x) -> p[1] * x + p[2])
tied = ParaMap(1, (q, x) -> twoscales([q[1], q[1]], x))
tied([3.0], 2.0) == 9.0                                # true

import Mathlib
open CategoryTheory MonoidalCategory
-- a morphism of Para(C): a parameter object and a map P ⊗ A ⟶ B
structure ParaHom {C : Type*} [Category C] [MonoidalCategory C] (A B : C) where
  P : C
  f : P ⊗ A ⟶ B
 
def ParaHom.comp {C : Type*} [Category C] [MonoidalCategory C] {A B D : C}
    (φ : ParaHom A B) (ψ : ParaHom B D) : ParaHom A D where
  P := ψ.P ⊗ φ.P
  f := (α_ ψ.P φ.P A).hom ≫ (ψ.P ◁ φ.f) ≫ ψ.f

-- Para over (Hask, (,)): parameters multiply under composition
newtype Para p a b = Para { run :: p -> a -> b }
 
(>>>) :: Para p a b -> Para q b c -> Para (q, p) a c
Para f >>> Para g = Para (\(q, p) a -> g q (f p a))
 
reparam :: (q -> p) -> Para p a b -> Para q a b          -- a 2-cell of Para
reparam alpha (Para f) = Para (f . alpha)
 
scale :: Para Double Double Double
scale = Para (*)
tied :: Para Double Double Double                           -- weight tying via the diagonal
tied = reparam (\w -> (w, w)) (scale >>> scale)
-- run tied 3 2 == 18