Several learned relations on the same variables can be pooled: add their fields, and the answers of the sum are the configurations that satisfy all of them, e.g. the intersection points of a learned circle and a learned line. This is a product of experts, read as a relation. It is exact at zero noise and slightly biased at the noise levels the learner works at, the same bias a single relation already has. Factors may see different variables, with a separator; sampling a product needs corrector steps and reweighting. Implemented as
ProductRelation(product).
Sources: original to this vault (design and analysis); Hinton, Training Products of Experts by Minimizing Contrastive Divergence, Neural Computation 2002; Du et al., Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023; code:
product.jl,answers.jlBibliography: Hinton 2002 · Du et al. 2023
Theory (CT-ML wiki): Markov Category
1. The idea
An implicit learner turns a dataset of joint configurations into a relation: the set of stable roots of its field , which is (for an ideal model) the gradient of a smoothed negative log-density (Implicit Diffusion Learners). Two such learners trained on different data over the same variables define two relations . Their intersection is the set of configurations both datasets allow, and it is found by solving
the field of the product density . Nothing is retrained: the intersection of two relations is a query on two models that already exist.
Examples: a circle and a line (two points), a circle and an ellipse (four points), a learned dynamics constraint and a learned measurement model (the states consistent with both), or two experts trained on different sensors.
2. Conditioning is a special case
The learner’s soft clamp, the residual , is already a product: the
second term is the field of an axis-aligned Gaussian centred at the anchor , with
precision . A hard clamp () is the limit, a Dirac on the clamped
coordinates. So every query was always an intersection, of the learned relation with the
relation “these coordinates have these values”. ProductRelation lets the second factor be
learned as well.
3. What the weights mean
is a tempering exponent: how strongly factor pulls. On an exact intersection both fields vanish, so the weights barely move the answer, only its conditioning and the Laplace covariance (the Jacobian is ). They matter when the relations do not meet: the stable roots are then compromises, nearer the factor with the larger weight. Nothing flags such a compromise yet. The residual norms of the separate factors at the root would be the natural diagnostic (Open Problems in Implicit Diffusion Learning T4).
4. Exactness: the obstruction at
Noising does not commute with products, , so the sum of fields at a noise level is not the field of the smoothed product (T9; Language Models §7). For sampling this is serious, and Du et al. correct it with MCMC steps at each level. For root finding it is much milder:
- the field’s noise levels are small, and the learner’s answers are already biased by the smoothing for a single relation;
- the product inherits a bias of the same size. Measured with closed-form models, all intersections are found, within 0.022 (circle ∩ line) and 0.029 (circle ∩ ellipse) of the exact points;
- the bias should shrink with the noise levels, as it does for a single relation (expected, not yet measured for products).
For sampling, the remedies are built (product §5): Langevin or MALA corrector steps at each level (Du et al.) fix the shape of the modes but not the mass between them, and sequential Monte Carlo reweighting over the noise levels (Skreta et al. 2025) fixes the mass. On circle ∩ line the upper intersection gets 0.655 of the samples with summed scores, 0.694 with Langevin correctors and 0.744 ± 0.019 with reweighting, against 0.751 exact.
The adjoint is exact for the field that is solved, so training through an intersection is unaffected (checked against finite differences, product §3).
5. Geometry: transversal and tangent intersections
Where the relations cross transversally, is well conditioned and the answer is sharp. Where they touch, it is nearly singular along the common tangent. The Laplace covariance then becomes long in that direction, which is the answer: the data barely determine the point. This is the composite version of the branch points of a single relation (T3).
6. Scope
- Different variable sets are covered:
relation => coordsplaces a factor on a subset of the joint, and a separator (, weight on the shared coordinates) prevents double counting of the shared marginal (product §4). The separator is trained, not derived. The full factor-graph composition, with messages between factors rather than one pooled field, is Messages are Inversions and The Diffusion Factor §4. - Logic beyond “and”. Union is a mixture of the densities, not a sum of fields; negation () is a difference of fields and has no stable roots where is flat. See Belief Algebra for the corresponding operations on beliefs.
A worked example with three trained networks (circle, line, ellipse), their intersections, Laplace covariances and fine-tuning through an intersection is the documentation’s tutorial Intersections: several learned relations at once.
Related: product, Implicit Diffusion Learners, Inference Signatures, Open Problems in Implicit Diffusion Learning, Belief Algebra, Kernel Methods for Implicit Learning