API

LenticulumLM.LenticulumLM — Module
LenticulumLM

A diffusion language model assembled from implicit learners. A sequence of tokens is a point of a joint space, one block of coordinates per position; windows of consecutive positions are implicit relations (Lenticulum.jl's diffusion learners), and the sequence is their product, joined by separators on the overlaps. One model then answers every query by root finding, whichever tokens are given: continuation, backward prediction and infilling are the same query with different clamps.

What is here so far

Exact against an $n$-gram chain's closed form; learned windows, a discrete learner and scale are the open part (Open Problems in Assembling Language Models.md).

source

Tokens

LenticulumLM.Vocabulary — Type
Vocabulary(symbols; scale = 1.0)

A finite vocabulary and its codes: symbol a is the vector $C\,e_a \in \mathbb R^V$, with $C$ = scale. A sequence of $L$ tokens is a point of $\mathbb R^{VL}$, position $i$ owning the coordinates block(v, i).

v = Vocabulary(['a', 'b', 'c'])
z = encode(v, ['a', 'c'])            # [1, 0, 0, 0, 0, 1]
decode(v, z)                         # ['a', 'c']
source
LenticulumLM.block — Method
block(v::Vocabulary, i) -> UnitRange

The coordinates of position i in the joint space.

source
LenticulumLM.codes — Method
codes(v::Vocabulary) -> Matrix

The $V \times V$ matrix whose column $a$ is the code of symbol $a$: $C$ times the identity.

source
LenticulumLM.decode — Method
decode(v::Vocabulary, z) -> Vector

The symbol of each position: the largest coordinate of its block, i.e. the nearest code.

source
LenticulumLM.encode — Method
encode(v::Vocabulary, tokens) -> Vector{Float64}

The point of $\mathbb R^{VL}$ that holds tokens, each as its code in its position's block. An entry nothing (an unknown token) leaves its block at zero, the centre of the codes.

source
LenticulumLM.nsymbols — Method
nsymbols(v::Vocabulary) -> Int

The number of symbols $V$, which is also the number of coordinates per position.

source
LenticulumLM.sequence_query — Method
sequence_query(v::Vocabulary, tokens) -> (z₀, ρ)

The query of a partly known sequence: tokens holds a symbol at each known position and nothing at each unknown one. The known positions are clamped (precision $\infty$), the unknown ones free (precision 0), and z₀ holds the known codes. Pass both to implicit_infer, implicit_mixture or position_marginals.

z₀, ρ = sequence_query(v, ['a', nothing, 'c'])      # infill the middle
source

Sequences assembled from relations

LenticulumLM.position_beliefs — Method
position_beliefs(v::Vocabulary, P) -> Vector{CategoricalBelief}

The columns of position_marginals' matrix as LenticulumCore CategoricalBeliefs labelled with the vocabulary's symbols: the per-position beliefs a factor graph passes along.

source
LenticulumLM.position_marginals — Method
position_marginals(v::Vocabulary, mix, ρ) -> (positions, P)

The answers of a sequence query read per position: mix is the MixtureBelief of implicit_mixture over the free coordinates (its default), ρ the query's precisions. Each answer is decoded to its nearest codes, and its weight is added to the symbol it gives at every free position. Returns the free positions and a $V \times$ (number of free positions) matrix P whose columns are categorical distributions.

source
LenticulumLM.sequence_relation — Method
sequence_relation(v::Vocabulary, window, separator, L) -> (relation, ps, st)

The relation of a sequence of L tokens assembled from one window relation, applied at every offset (shared parameters, as a stationary model shares them), and one separator on every overlap of two consecutive windows, with weight $-1$:

\[p(x_1, \dots, x_L) = \frac{\prod_{k=1}^{L-n+1} p_{\mathrm{win}}(x_k, \dots, x_{k+n-1})} {\prod_{k=2}^{L-n+1} p_{\mathrm{sep}}(x_k, \dots, x_{k+n-2})} .\]

window and separator are (relation, ps, st) triples, e.g. from window_relation of an $n$-gram table and of its $(n-1)$-gram marginal. separator = nothing leaves the separators out (and so counts each overlap's marginal once per window that covers it). The result is a ProductRelation with its parameter and state tuples.

source
LenticulumLM.token_marginals — Method
token_marginals(v::Vocabulary, relation, ps, st, tokens; nstarts = 16V^u, spread = 1.0, kwargs...)
    -> (positions, P, mix)

Answer a partly known sequence: tokens holds a symbol or nothing at each position (sequence_query). Every answer of the query is found by implicit_mixture (from nstarts starts, by default 16 per possible completion of the $u$ unknown positions; kwargs go to it) and read per position by position_marginals. The same relation answers a continuation ([a, b, nothing]), a backward prediction ([nothing, b, c]) and an infilling ([a, nothing, c]).

source
LenticulumLM.window_relation — Method
window_relation(v::Vocabulary, P; s = 0.05, schedule = VPSDE(), nodes = nothing) -> (relation, ps, st)

The closed-form implicit relation of an $n$-gram table: P is an $n$-dimensional array of size $V \times \cdots \times V$ whose entry P[a₁, …, aₙ] is the probability of the window $(a_1, \dots, a_n)$. The relation is a GaussianMixtureEps with one component of width s at the codes of every window with positive probability, weighted by it. It is the relation a diffusion model trained on these windows would converge to, with every derivative in closed form. nodes default to field_nodes with 8 antithetic pairs per level; the field is scaled by density_lambda, so its energy is a calibrated negative log-density.

source