design theory

Several learned relations on the same variables can be pooled: add their fields, and the answers of the sum are the configurations that satisfy all of them, e.g. the intersection points of a learned circle and a learned line. This is a product of experts, read as a relation. It is exact at zero noise and slightly biased at the noise levels the learner works at, the same bias a single relation already has. Factors may see different variables, with a separator; sampling a product needs corrector steps and reweighting. Implemented as ProductRelation (product).

Sources: original to this vault (design and analysis); Hinton, Training Products of Experts by Minimizing Contrastive Divergence, Neural Computation 2002; Du et al., Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023; code: product.jl, answers.jl

Bibliography: Hinton 2002 · Du et al. 2023

Theory (CT-ML wiki): Markov Category

1. The idea

An implicit learner turns a dataset of joint configurations into a relation: the set of stable roots of its field , which is (for an ideal model) the gradient of a smoothed negative log-density (Implicit Diffusion Learners). Two such learners trained on different data over the same variables define two relations . Their intersection is the set of configurations both datasets allow, and it is found by solving

the field of the product density . Nothing is retrained: the intersection of two relations is a query on two models that already exist.

Examples: a circle and a line (two points), a circle and an ellipse (four points), a learned dynamics constraint and a learned measurement model (the states consistent with both), or two experts trained on different sensors.

2. Conditioning is a special case

The learner’s soft clamp, the residual , is already a product: the second term is the field of an axis-aligned Gaussian centred at the anchor , with precision . A hard clamp () is the limit, a Dirac on the clamped coordinates. So every query was always an intersection, of the learned relation with the relation “these coordinates have these values”. ProductRelation lets the second factor be learned as well.

3. What the weights mean

is a tempering exponent: how strongly factor pulls. On an exact intersection both fields vanish, so the weights barely move the answer, only its conditioning and the Laplace covariance (the Jacobian is ). They matter when the relations do not meet: the stable roots are then compromises, nearer the factor with the larger weight. Nothing flags such a compromise yet. The residual norms of the separate factors at the root would be the natural diagnostic (Open Problems in Implicit Diffusion Learning T4).

4. Exactness: the obstruction at

Noising does not commute with products, , so the sum of fields at a noise level is not the field of the smoothed product (T9; Language Models §7). For sampling this is serious, and Du et al. correct it with MCMC steps at each level. For root finding it is much milder:

  • the field’s noise levels are small, and the learner’s answers are already biased by the smoothing for a single relation;
  • the product inherits a bias of the same size. Measured with closed-form models, all intersections are found, within 0.022 (circle ∩ line) and 0.029 (circle ∩ ellipse) of the exact points;
  • the bias should shrink with the noise levels, as it does for a single relation (expected, not yet measured for products).

For sampling, the remedies are built (product §5): Langevin or MALA corrector steps at each level (Du et al.) fix the shape of the modes but not the mass between them, and sequential Monte Carlo reweighting over the noise levels (Skreta et al. 2025) fixes the mass. On circle ∩ line the upper intersection gets 0.655 of the samples with summed scores, 0.694 with Langevin correctors and 0.744 ± 0.019 with reweighting, against 0.751 exact.

The adjoint is exact for the field that is solved, so training through an intersection is unaffected (checked against finite differences, product §3).

5. Geometry: transversal and tangent intersections

Where the relations cross transversally, is well conditioned and the answer is sharp. Where they touch, it is nearly singular along the common tangent. The Laplace covariance then becomes long in that direction, which is the answer: the data barely determine the point. This is the composite version of the branch points of a single relation (T3).

6. Scope

  • Different variable sets are covered: relation => coords places a factor on a subset of the joint, and a separator (, weight on the shared coordinates) prevents double counting of the shared marginal (product §4). The separator is trained, not derived. The full factor-graph composition, with messages between factors rather than one pooled field, is Messages are Inversions and The Diffusion Factor §4.
  • Logic beyond “and”. Union is a mixture of the densities, not a sum of fields; negation () is a difference of fields and has no stable roots where is flat. See Belief Algebra for the corresponding operations on beliefs.

A worked example with three trained networks (circle, line, ellipse), their intersections, Laplace covariances and fine-tuning through an intersection is the documentation’s tutorial Intersections: several learned relations at once.

Related: product, Implicit Diffusion Learners, Inference Signatures, Open Problems in Implicit Diffusion Learning, Belief Algebra, Kernel Methods for Implicit Learning