What an implicit diffusion learner can represent. For a fixed node set, the stable answers of an energy network’s learner approximate the stable answers of any target energy whose answers are nondegenerate, uniformly over a compact set of queries; an ε-network’s learner does the same for any target vector field. The answers move by at most , where is the distance of the residuals and measures how well-conditioned the target’s answers are. The theorem is about expressiveness. Whether training reaches such parameters is a separate question, and §6 lists what stays open.
The diffusion counterpart of Universal Approximation by Nash-Tognoli.
Sources: original to this vault (statement and proofs, not peer reviewed); uses Pinkus, Approximation Theory of the MLP Model in Neural Networks, Acta Numerica 1999 (Theorem 4.1, density in ); Hornik, Approximation Capabilities of Multilayer Feedforward Networks, Neural Networks 1991; for §6, Genovese, Perone-Pacifico, Verdinelli & Wasserman, Nonparametric ridge estimation, Annals of Statistics 2014
Bibliography: Pinkus 1999 · Hornik 1991 · Genovese et al. 2014
1. The objects
The learner of Implicit Diffusion Learners has fixed nodes , , weight , and the field
For an energy network, , the field is with
(energy_network §3). A query clamps the coordinates at and anchors the free coordinates (with ) at with precisions . Write for its data and for the unknowns. Its residual is
and an answer is a zero of . It is stable if
, the flag stable of implicit_infer. For an
energy network that is a strict local minimum of .
A target is any vector field , or for energy networks with . The query turns it into a residual in the same way.
Regularity. Let (query data) and (an answer window) be compact. A residual , on an open set , is regular on if
- (R1) at every zero , is invertible and ;
- (R2) for and .
(R1) says every answer is nondegenerate and is clearly stable () or clearly unstable (). Only the smallest eigenvalue of enters: the constant of Lemma 1 controls it and , and nothing else is used. (An earlier version asked that have no zero eigenvalue at all, which does not control at an unstable answer; review of 2026-11-10.) For a gradient, is the Hessian block and (R1) says that every critical point in the fibre is nondegenerate (a Morse condition on the fibres). It excludes branch points (§5).
2. Answers persist
Lemma 1 (persistence of answers). Let be regular on . There are a compact neighbourhood of and constants , such that for every with
and every , the zeros of in and those of in are in a bijection that moves no zero by more than and preserves stability. Moreover is regular on with constant .
Here over the zeros of in of , and denotes the operator 2-norm.
Proof. Let ; it is compact, and because and are continuous and positive in absolute value on by (R1). By (R2), . Choose so small that and, by uniform continuity of on , whenever and .
Step 1: one zero per ball. Fix , , , and let . For , with , . So and has at most one zero in . (With : distinct zeros of are at least apart.)
Step 2: existence. has on , and . With , maps into itself () and contracts, so it has a fixed point : and .
Step 3: stability is kept. with . By Weyl’s inequality : the sign is the same, and both and are at least .
Step 4: no other zeros. The set is closed: its complement in is open, because by the implicit function theorem ( invertible, interior to ) each zero continues to nearby . So is compact, on it, and . For , every zero of in lies within of a zero of , and by Step 1 it is the of Step 2 for that .
Take . Then , so different (at least apart) get different ; , so ; and by Step 4 every zero of in is reached.
The constant is effective: an answer moves by at most , where is the worst conditioning of the target’s answers. Near a branch point (§5), and so does the guarantee.
3. The node sum can realise any target
The network enters only through the node sum. The question is whether every target energy is for some . It is, on a compact set, even though the nodes of one level see the network at shifted points (antithetic pairs see it at ).
Lemma 2 (inverting a finite sum of shifts). Let be distinct, , and with compact support. There is with
Proof. Pick with the values pairwise distinct (avoid finitely many hyperplanes) and number the shifts so that is the largest. Let and . Define on and
elsewhere. The points lie at least lower in the direction , so by induction over the slabs , is well defined there and equals a finite linear combination of translates of . It is therefore . On the formula also gives , since there and the other terms lie lower still. So the formula holds everywhere, and at it is the claimed identity.
Proposition 3 (exact realisation). Let be compact and fix a level of the nodes. For every there is with on , and at every other node level. For every there is a noise predictor with on .
Proof (energy). Let and . Group equal shifts, adding their into . Take a smooth cutoff on with compact support, and apply Lemma 2 to . Set , so that , and make at the other levels with a smooth factor in . Then on . The field case is the same with , weights and .
With noisefree_nodes (, one node) this is just . With
a single antithetic pair (, ) it is an alternating sum,
, whose norm on a compact set grows at most
linearly in the number of slabs . For equal-weight shifts
(), the slab recurrence has amplification factor per slab of width
, so on is finite for any fixed (finitely many slabs intersect )
but its worst-case bound from the slab recurrence grows exponentially as .
(Conjecture / unverified expectation: whether generic shift configurations with admit
another smooth inverse with polynomial -norm growth as is not proved here;
approximation does not need it. The unshifted rescaling has
, and with
and on a neighbourhood of , uniform continuity of makes this
in as ; that suffices for -approximation, bypassing exact shift
deconvolution. A rate needs more: if , and only if also the shifts
are centred, , and . Without centring the first-order term remains:
shifts with equal weights and give error .)
4. The theorem
Theorem 4 (universal approximation of answers). Fix the nodes, , and the predictor’s time features , with for the node levels of some level . Let the network be (resp. ) with a one-hidden-layer network whose activation is (resp. ) and not a polynomial. Let finitely many queries (polarities and precisions), each with compact data and window , have regular target residuals for a target energy (resp. for a target field ). Then for every there are parameters such that, for every one of these queries and every , the answers of the learner in and of the target in are in a bijection that moves no answer by more than and preserves stability. In particular the learner’s stable answers are within Hausdorff distance of the target’s, uniformly in .
Proof. Take compact containing every with in the neighbourhoods of Lemma 1, for all the queries, and take the smallest among them. Proposition 3 gives with on , and it can be written as with on (a smooth factor in the feature vector that is at and at the other levels’ features). By Pinkus’ theorem, one-hidden-layer networks with a activation that is not a polynomial are dense in on compact sets (with all derivatives up to order ). So for every some is within of in on the compact set . Since , the chain rule gives . The residuals differ by (the precision term is the same), so their distance in is at most . Choose and apply Lemma 1. For fields, use and .
Remarks.
- Time features. The package’s sinusoidal features include the frequency , , which is injective on . So every level can be singled out.
- Depth. A deeper network with the same activation can emulate the shallow one: with
, in on compact sets as , so the
extra layers approximate the identity.
swish,tanhandsoftplusqualify. ReLU is not and is excluded, as it is for energy networks anyway. - Noise blocks. With held block levels the node inputs are with per-coordinate factors, and the weights change (implicit §9). The proof only uses that each node input is an invertible affine map of and that the weights are positive, so it carries over.
- ε-networks are universal for a larger class: any field with regular answers, gradient or not. Energy networks reach exactly the gradient targets. Both reach every relation that has an energy (§5).
5. Which relations are regular targets
The theorem is about targets with regular answers. Which relations have one?
- Functions. For , on a compact , take and the query : , , . With a ball containing in its interior, the learner’s single answer approximates uniformly on . Universal approximation of functions is the special case of one branch. Since Theorem 4 makes close to in , mixed derivatives included, the implicit function theorem also makes the derivative of the answer, , close to .
- Finitely many branches with . For disjoint branches over , take . The fibre derivative is a polynomial of degree with simple real roots: the minima (with ) and one maximum between each pair (Rolle). So (R1) holds, and (R2) holds for with large.
- The circle away from its branch points. , query , . The answers are stable with , and is unstable with . Then , so the guarantee degrades as the query approaches . This is T3 made quantitative. The same serves the query .
- Chains and products. A product of learners, one per factor, reaches every
target of the factorised form . Apply Theorem 4 to each
factor; the errors add.
examples/chain.jlshows such chains of three to five relations (product §4).
6. What the theorem does not say
| explicit (Weierstraß, networks) | algebraic (Nash–Tognoli) | implicit diffusion (Theorem 4) | |
|---|---|---|---|
| approximated object | continuous functions | compact smooth manifolds, up to diffeomorphism | the answers of any regular target, per query |
| in the data’s own space? | yes | not necessarily (Borel–Haefliger) | yes |
| effective? | yes (rates) | no | the persistence step yes (); the network step as effective as rates for networks |
| nondegenerate output? | n/a | singular fits are generic | answers within of a regular target are regular |
Gap 2 of Nash–Tognoli does not arise here: everything stays in the data’s coordinates, and smooth functions carry no algebraic-cycle obstruction. Gap 3 (singular fits) is answered near a regular target, because Lemma 1 makes the learner’s answers regular there as well. What remains open:
- Expressiveness is not learning. Training does not aim at . At its population optimum
the network is the true score, . The learner’s relation is then the
set of stable answers of the smoothed density’s , the ridge of
Implicit Diffusion Learners §5. That is not the data’s relation: the smoothing bias (T2)
adds up along chains (
examples/chain.jl). Theorem 4 does say something here. If is regular and the trained network is -close to on the query region, the answers are those of within . But score matching controls an error, not a error on the query region, and the two differ exactly where data are sparse (T5). Consistency is open: that the answers of approach the data’s relation as the levels go to and the data grow. Related results exist for ridges of kernel density estimates (Genovese et al. 2014). - Rates. Lemma 1 is effective, and the network step inherits whatever approximation rates hold for . Those rates depend on its smoothness and on the input dimension . The construction of Lemma 2 is not frugal: with antithetic shifts it sums about translates. In practice the natural target is smooth at the node levels and needs no deconvolution. Rates for the realistic target are open.
- Branch points (T3). (R1) excludes them, and the constant degrades like near them. Folds of the answer set are generic in families of functions and structurally stable, so a statement that the fold itself persists and moves by should hold. It is not proved here.
- Which relations are regular targets. §5 proves it for functions, for scalar multi-branch relations and for the circle. For a general compact submanifold and every polarity at once, it is open. with a defining submersion makes the answers on regular exactly where is a local graph over the clamped coordinates. But it needs a trivial normal bundle, and it does not control the other critical points of the fibres. This is the implicit counterpart of the question Nash–Tognoli answers for varieties.
- Finding the answers (T1, T11). The theorem gives existence and the count of answers per query, not that the solver finds all of them, or any of them, from a given start.
- Queries outside (T4, T5). Nothing is claimed off the compact query set. The learner may have answers there that the target lacks.
7. Multi-polarity queries and the codimension paradox (heuristic analysis and conjecture)
Status: heuristic analysis and conjecture (not proved). The three regimes below describe the expected geometry of the fibre Hessian across query polarities of different dimensions; making regime 2 rigorous as a theorem requires nondegeneracy assumptions on the smoothed in-manifold log-density as that remain open (see §6.1 and §6.4).
For any choice of free coordinates with , the fibre residual
is a square system of equations in unknowns. How can a single scalar potential yield isolated, regular answers () across queries with different numbers of free coordinates (for instance, MNIST classification image -> label with alongside conditional generation label -> image with )?
Suppose the data lie on a smooth submanifold of dimension and codimension . Three distinct geometric regimes govern the fibre Hessian :
- Transverse manifold slicing (, ). If is a pure tubular distance potential vanishing on , then at every the full Hessian has rank (positive definite on the normal space ) and corank (zero along the tangent space ). When and the -dimensional coordinate fibre intersects transversely (), the restriction is strictly positive definite (), so the intersection points are isolated regular answers.
- Conditional mode finding (, ). If were constant along and , the intersection would be an -dimensional continuum of roots and would have corank at least everywhere on , violating (R1). A diffusion learner’s potential , however, is not constant along : it carries the negative log-density of the data distribution on alongside the normal penalty . When , the roots of are the 0-dimensional local modes of the conditional density on the slice , where the in-manifold density curvature breaks the tangent degeneracy and makes .
- Soft-anchored projection (, ). When the free coordinates carry positive anchor precisions , adding makes even when is flat along , selecting the metric projection of the anchor onto the slice.
Related: Implicit Diffusion Learners, Open Problems in Implicit Diffusion Learning, Universal Approximation by Nash-Tognoli, energy_network, product, Implicit Learners