implementation

A language model assembled from implicit learners: one relation per window of consecutive tokens, joined by separators on the overlaps. With the windows of an -gram chain this is that chain exactly, and one assembled relation answers continuation, backward prediction and infilling, each to within of the closed form.

Sources: code: sequence.jl, tokens.jl; Hinton, Training Products of Experts by Minimizing Contrastive Divergence, Neural Computation 2002 (the product); Du et al., Reduce, Reuse, Recycle: Compositional Generation using Energy-Based Diffusion Models and MCMC, ICML 2023 (adding scores, and why it is not exact at )

Bibliography: Hinton 2002

Lenticulum (vault): product §4 (separators, longer chains) · Implicit Diffusion Learners · Language Models §7

1. The construction

A window relation sees consecutive positions. Applied at every offset and joined by a separator on each overlap of two consecutive windows,

When is the -gram distribution of a stationary -th order Markov chain and its -gram marginal, this is the chain’s joint distribution: the junction-tree factorisation of a path. sequence_relation builds it as a Lenticulum ProductRelation, with the window relation on the coordinates of positions and the separator, weight , on the overlap. The same relation object and parameters are used at every offset: the model is stationary, and its parameters are shared as a sliding network’s would be.

window_relation(v, P) is the closed-form window: a Gaussian mixture of width with one component at the codes of every window with , weighted by it. This is the relation a diffusion model trained on those windows would converge to, with every derivative exact, so it is the test oracle for the assembly. A learned window would replace it without changing anything else.

2. Reading the answers as tokens

A query clamps the known positions. implicit_mixture finds its stable roots, each with a Laplace weight , which is the probability of the completion it represents. position_marginals decodes every root to its nearest codes and adds its weight to the symbol it gives at each free position. The result is one categorical distribution per free position, the marginal of the query’s posterior. token_marginals does both steps, and position_beliefs turns the columns into LenticulumCore CategoricalBeliefs labelled with the symbols, the belief type a factor graph over tokens passes along.

The number of roots is the number of completions with positive probability, for free positions of a chain with full support. token_marginals starts 16 solves per completion by default. That is fine for the test’s three symbols and hopeless for a real vocabulary (§4).

3. Measured

A stationary bigram chain over three symbols, , its stationary , window table and separator . Width , the default nodes. Errors are the largest difference of a marginal from its closed form:

queryclosed formanswerserrorwithout separatorstime
3[a, ?, ?], last position90.0138.7 s (first call, compiling)
3[?, ?, c], first position90.0190.7 s
3[a, ?, c]30.0420.2 s
4[a, ?, ?, ?], last position270.0215.5 s
4[?, ?, ?, c], first position270.0303.3 s
4[a, ?, ?, c], position 290.0601.1 s

Every completion is found, and every query direction is answered by the same relation. Without the separators each inner position’s marginal counts once per window covering it, and the error grows with the number of inner positions (Lenticulum’s product.md §4 measures the same for chains of three to five binary relations).

4. What it does not do

  • The windows are not learned yet. window_relation is the exact relation of a known table. A learned window (an energy network on coordinates, trained by denoising on the windows of a corpus) plugs into sequence_relation unchanged. Training one, and measuring how its smoothing bias adds up along a sequence, is the next step (Open Problems in Assembling Language Models L2).
  • The separators are supplied. They must be the marginals of the windows on the overlaps, or the product is not the chain. For a learned window that means a second learned relation, or the window model queried with the other positions marginalised away.
  • Enumerating answers is exponential. A query with free positions has up to roots. The categorical marginals here sum over all of them. A real model needs per-position beliefs passed between windows (message passing on the chain) instead of the joint answers, which means categorical beliefs over continuous codes (Open Problems in Assembling Language Models L5).
  • Composition is exact here only because the windows are closed form. For networks the product of smoothed relations is not the smoothed product (the obstruction of Lenticulum’s Language Models §7), and the error is expected to add up along the chain like the smoothing bias does.

Related: tokens, Assembling a Language Model from Implicit Learners, Open Problems in Assembling Language Models