implementation

Products of relations. Several implicit learners pooled: the field of the product is the weighted sum of their fields, so its stable roots are where all the relations hold at once, e.g. the two points where a learned circle meets a learned line. Factors may see different variables, with a separator term against double counting. ProductRelation is one more AbstractImplicitRelation, so every query function works on it unchanged, gradients flow back into each factor, and it can be sampled, with corrector steps and reweighting.

Sources: code: product.jl, sampler.jl, implicit.jl (AbstractImplicitRelation, _params_cotangent), answers.jl, examples/product_sampling.jl; Hinton, Training Products of Experts by Minimizing Contrastive Divergence, Neural Computation 2002; Du et al., Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023 (adding scores, and why it is not exact at ); Roberts & Tweedie, Exponential Convergence of Langevin Distributions and Their Discrete Approximations, Bernoulli 1996 (MALA); Del Moral, Doucet & Jasra, Sequential Monte Carlo Samplers, JRSS B 2006; Skreta et al., Feynman-Kac Correctors in Diffusion, ICML 2025 (reweighting a product of experts)

Bibliography: Hinton 2002 · Du et al. 2023 · Roberts & Tweedie 1996 · Del Moral et al. 2006 · Skreta et al. 2025

Theory: Composing Diffusion Factors · Implicit Diffusion Learners · Open Problems in Implicit Diffusion Learning

1. The construction

An implicit learner’s field is, for an ideal predictor, the gradient of a smoothed negative log-density. A weighted sum of such fields is therefore the field of the product of the densities:

ProductRelation(m₁, m₂, …; weights = β) implements exactly the three functions the solvers use:

functionproduct
prior_field
prior_jacobian
_params_cotangenta tuple: factor gets the adjoint

Parameters and states are tuples, one entry per factor: ps = (ps₁, ps₂). Passing a single parameter set raises an ArgumentError. Products nest, because a product is itself an AbstractImplicitRelation.

2. What works on it unchanged

  • implicit_infer: one answer, with clamps (ρ = Inf) and soft anchors as usual;
  • implicit_roots: all answers by multi-start, e.g. the intersection points;
  • implicit_laplace: a covariance per answer;
  • implicit_pullback: the IFT adjoint. The cotangent of the parameters is a tuple, so a loss on the intersection trains every factor;
  • implicit_energy, and ordering roots by energy, when every factor is an EnergyNetwork or a closed-form mixture (the energy of the product is ), and so implicit_mixture with Laplace mass weights;
  • implicit_sample, when all factors share a noise schedule (§5).

The refactor that made this possible is small. implicit.jl now types its functions with the abstract supertype, and moves the parameter loop of the pullback into _params_cotangent, the one place where a product differs.

3. Accuracy: the smoothing bias

Pooling is exact only at zero noise. At the field’s noise levels the product of the smoothed densities is not the smoothed product, (Open Problems in Implicit Diffusion Learning T9). For root finding this produces the same kind of bias that a single relation already has (its ridge sits slightly on the inside of a bend), not a failure. Closed-form mixtures (GaussianMixtureEps, ), the default field_nodes:

productanswers foundexactlargest distance to exact
circle ∩ line 220.022
circle ∩ ellipse (1.5, 0.6)440.029

The adjoint is exact for the field that is actually solved. Clamping on circle ∩ ellipse, is −0.65669290204 by the adjoint and −0.65669290206 by finite differences. A parameter of the ellipse gives 0.0220369733 by both.

4. Different variables: separators

A factor may see a subset of the joint coordinates: relation => coords. Its field is evaluated on z[coords] and added back at those positions, and the same holds for the Jacobian block, the JVP and VJP, the parameter cotangent and the energy. Relations on overlapping variables compose as in a junction tree,

exact when and both factors have the marginal on . The separator is a relation on the shared coordinates with weight :

ProductRelation(mA => [1, 2], mB => [2, 3], mS => [2]; weights = [1, 1, -1])

Without it the shared marginal is counted once per factor, instead of : it is sharpened, and its modes are reweighted. The constructor requires every coordinate to be seen by some relation, and the weights of the relations that see a coordinate to sum to a positive number, so a separator cannot outweigh its factors.

Measured. A chain with two blobs at and , weights 0.8 and 0.2 (closed-form mixtures of width 0.05, density_lambda), implicit_mixture with no coordinate clamped:

main blobsecond blobsd of other stable roots
with separator0.8000.2000.133one, weight
without0.9410.0590.094two, weight

: without the separator the weights are those of , and is narrower by . The extra roots are compromises such as , where and disagree about . They are stable but carry no mass. Division by the separator’s small density in the gap is what makes them stable, and it matters for sampling (§5).

The adjoint through such a chain (curves and with a separator, clamped) matches central differences to for the clamp and for a parameter of .

Longer chains (examples/chain.jl). relations with separators on the inner variables,

so for the outer relations share no variable and a clamp on reaches only through the inner ones. Two problems with known answers, :

A telephone chain. A bit passed along links, each flipping it with probability 0.2 (from ) or 0.3 (from ). Relation is a mixture of blobs at weighted ; the separators are the marginals . With held, implicit_mixture (40·2^K starts) returns every path of the bit as an answer:

paths found with separatorsexact largest path-weight errorwithout separatorstime
38 of 80.65000.65000.6863 (error 0.062)8 s
416 of 160.62500.62500.6785 (error 0.097)6 s
532 of 320.61250.61250.6776 (error 0.123)13 s

Every path’s Laplace weight is its probability . Without the separators each inner marginal counts twice, and the error grows with the number of inner variables.

A chain of curves. Relation is the curve , , as 65 points; is held. The answer approximates , and the adjoint’s is compared with central differences of the solve and with the true derivative :

levelslargest bias adjointcentral differences
30.002–0.050.0440.81740.81740.8673
40.002–0.050.0880.67630.67630.6145
50.002–0.050.1340.60220.60220.3696
30.0005–0.0050.0060.84550.84550.8673
40.0005–0.0050.0170.59170.59170.6145
50.0005–0.0050.0320.36070.36070.3696

The adjoint is exact for the smoothed relation at every length. The smoothing bias (Implicit Diffusion Learners §5) adds up along the chain, about linearly in , and so does the error of the derivative. At ten times lower noise levels both shrink, the bias by about a factor of 8 at , and the derivative comes within 3% of the true one at .

5. Sampling a product

implicit_sample runs the reverse diffusion with the summed noise prediction , the summed score. At that is the score of , not of the noised product (T9). For root finding the error is a small bias (§3). For sampling it is a wrong distribution: the reverse process does not end at the product. Two remedies, both options of implicit_sample:

  • Corrector steps (corrector = :langevin or :mala, Du et al.): after every reverse step, corrector_steps Langevin steps with , which move the chains towards at the current level. :mala adds the Metropolis–Hastings test, which needs log-densities.
  • Reweighting (smc = true): a sequential Monte Carlo sampler for the sequence . The ancestral step is the proposal , the forward noising the backward kernel, and each chain’s weight is multiplied by . Chains are resampled (systematically) when the effective sample size drops below half, and at the end. Moves between reweightings must leave invariant, so it combines with :mala, not with :langevin. This is the discrete-time form of the Feynman–Kac correctors of Skreta et al.

Measured (examples/product_sampling.jl). Two products with exact answers: the chain of §4, and circle (upper half three times heavier) ∩ line on shared variables, whose exact product puts 0.751 on the upper intersection. 100 steps, 2000 chains, mean ± sd over six seeds; “off” is the share of samples near no mode, “spread” the standard deviation within the main mode (exact 0.050), time per run in seconds:

samplerchain: main (0.800)offcircle: main (0.751)off (0.008)spreadtime
plain0.736 ± 0.0140.1460.655 ± 0.0110.0810.0870.5
Langevin ×3, 0.766 ± 0.0090.0950.662 ± 0.0070.0060.0502.2
Langevin ×10, 0.788 ± 0.0080.0030.694 ± 0.0050.0060.0495.3
MALA ×10, 0.780 ± 0.0030.0410.695 ± 0.0120.0070.04910.7
SMC0.813 ± 0.0350.0000.760 ± 0.0190.0050.0490.9
SMC + MALA ×3, 0.798 ± 0.0130.0000.744 ± 0.0190.0060.0504.0

What this shows:

  • Summed scores alone are wrong in both ways. On the circle, 8% of samples lie off both intersections and the spread is 0.087. On the chain, 15% land in the compromise states of §4.
  • Corrector steps fix the shape, not the mass. A few Langevin steps bring the spread to 0.050 and the strays to the exact level. The split between modes stays wrong (0.69 against 0.75): by the time the modes are separated, local moves cannot carry mass across.
  • Reweighting fixes the mass. SMC alone gets both splits right within its seed-to-seed spread, at twice the cost of the plain sampler, and removes the compromise states. MALA moves between reweightings reduce the variance on the chain (0.013 against 0.035) by refreshing the resampled duplicates.
  • Langevin and MALA perform about the same as correctors. MALA’s test removes the discretisation bias inside a mode, which is small here.

Both remedies need what plain sampling does not. :mala and smc need log-densities: mixtures, kernel estimates and energy networks have them, a direct ε-network does not. SMC’s weights degenerate when the proposal is poor; there are only 2000 chains and two modes here, and its cost in higher dimensions is not measured.

6. Limits

  • The separator is supplied, not derived. It is a relation the user trains on the shared marginal (or a kernel estimate of it). Deriving it from a factor would need the factor’s marginal score, which a joint diffusion model does not give in closed form.
  • Tangent intersections are ill-conditioned: the Jacobian of the sum is nearly singular, so the Laplace covariance blows up along the common tangent. This is the right answer, but slow to converge.
  • Empty intersections. If the relations do not meet, the stable roots are compromises between them, weighted by β. They are still returned; nothing flags them yet.
  • Sampling with clamps under smc holds the clamped coordinates at without noise, so the targets are the conditionals at each level; it does not combine with RePaint’s resample.

7. Tests

test/runtests.jl, “products of relations: intersections”:

  • the field equals the weighted sum;
  • a non-tuple ps is rejected;
  • two and four roots within 0.05 of the exact intersections;
  • the adjoint for the clamp and for one ellipse parameter matches central differences (rtol 1e-6);
  • the parameter cotangent is a tuple.

“products over different variables: separators”: the field, Jacobian blocks, JVP and VJP of a chain against central differences; the Laplace weights with and without the separator; the adjoint through a chain of curves; the telephone chain of three relations (all 8 paths, their weights and within of the exact ones); the energy; the constructor’s checks.

“sampling a product: corrector steps and reweighting”: circle ∩ line, plain (strays above 5%), Langevin (strays below 1%, spread 0.05), SMC + MALA (upper share within 0.04 of 0.75); SMC on a single clamped model; the refused combinations.