API
LenticulumLM.LenticulumLM — Module
LenticulumLMA diffusion language model assembled from implicit learners. A sequence of tokens is a point of a joint space, one block of coordinates per position; windows of consecutive positions are implicit relations (Lenticulum.jl's diffusion learners), and the sequence is their product, joined by separators on the overlaps. One model then answers every query by root finding, whichever tokens are given: continuation, backward prediction and infilling are the same query with different clamps.
What is here so far
tokens.jl—Vocabulary: tokens as scaled one-hot codes, one block of coordinates per position;encode,decode, andsequence_query, which turns a partly known sequence into clamps.sequence.jl—window_relation(the closed-form relation of an $n$-gram table),sequence_relation(windows and separators assembled into one relation), andtoken_marginals/position_marginals/position_beliefs(every answer of a query, read as a categorical belief per position).
Exact against an $n$-gram chain's closed form; learned windows, a discrete learner and scale are the open part (Open Problems in Assembling Language Models.md).
Tokens
LenticulumLM.Vocabulary — Type
Vocabulary(symbols; scale = 1.0)A finite vocabulary and its codes: symbol a is the vector $C\,e_a \in \mathbb R^V$, with $C$ = scale. A sequence of $L$ tokens is a point of $\mathbb R^{VL}$, position $i$ owning the coordinates block(v, i).
v = Vocabulary(['a', 'b', 'c'])
z = encode(v, ['a', 'c']) # [1, 0, 0, 0, 0, 1]
decode(v, z) # ['a', 'c']LenticulumLM.block — Method
block(v::Vocabulary, i) -> UnitRangeThe coordinates of position i in the joint space.
LenticulumLM.codes — Method
codes(v::Vocabulary) -> MatrixThe $V \times V$ matrix whose column $a$ is the code of symbol $a$: $C$ times the identity.
LenticulumLM.decode — Method
decode(v::Vocabulary, z) -> VectorThe symbol of each position: the largest coordinate of its block, i.e. the nearest code.
LenticulumLM.encode — Method
encode(v::Vocabulary, tokens) -> Vector{Float64}The point of $\mathbb R^{VL}$ that holds tokens, each as its code in its position's block. An entry nothing (an unknown token) leaves its block at zero, the centre of the codes.
LenticulumLM.nsymbols — Method
nsymbols(v::Vocabulary) -> IntThe number of symbols $V$, which is also the number of coordinates per position.
LenticulumLM.sequence_query — Method
sequence_query(v::Vocabulary, tokens) -> (z₀, ρ)The query of a partly known sequence: tokens holds a symbol at each known position and nothing at each unknown one. The known positions are clamped (precision $\infty$), the unknown ones free (precision 0), and z₀ holds the known codes. Pass both to implicit_infer, implicit_mixture or position_marginals.
z₀, ρ = sequence_query(v, ['a', nothing, 'c']) # infill the middleLenticulumLM.symbol_index — Method
symbol_index(v::Vocabulary, s) -> IntThe index of symbol s (its code is column symbol_index(v, s) of codes).
Sequences assembled from relations
LenticulumLM.position_beliefs — Method
position_beliefs(v::Vocabulary, P) -> Vector{CategoricalBelief}The columns of position_marginals' matrix as LenticulumCore CategoricalBeliefs labelled with the vocabulary's symbols: the per-position beliefs a factor graph passes along.
LenticulumLM.position_marginals — Method
position_marginals(v::Vocabulary, mix, ρ) -> (positions, P)The answers of a sequence query read per position: mix is the MixtureBelief of implicit_mixture over the free coordinates (its default), ρ the query's precisions. Each answer is decoded to its nearest codes, and its weight is added to the symbol it gives at every free position. Returns the free positions and a $V \times$ (number of free positions) matrix P whose columns are categorical distributions.
LenticulumLM.sequence_relation — Method
sequence_relation(v::Vocabulary, window, separator, L) -> (relation, ps, st)The relation of a sequence of L tokens assembled from one window relation, applied at every offset (shared parameters, as a stationary model shares them), and one separator on every overlap of two consecutive windows, with weight $-1$:
\[p(x_1, \dots, x_L) = \frac{\prod_{k=1}^{L-n+1} p_{\mathrm{win}}(x_k, \dots, x_{k+n-1})} {\prod_{k=2}^{L-n+1} p_{\mathrm{sep}}(x_k, \dots, x_{k+n-2})} .\]
window and separator are (relation, ps, st) triples, e.g. from window_relation of an $n$-gram table and of its $(n-1)$-gram marginal. separator = nothing leaves the separators out (and so counts each overlap's marginal once per window that covers it). The result is a ProductRelation with its parameter and state tuples.
LenticulumLM.token_marginals — Method
token_marginals(v::Vocabulary, relation, ps, st, tokens; nstarts = 16V^u, spread = 1.0, kwargs...)
-> (positions, P, mix)Answer a partly known sequence: tokens holds a symbol or nothing at each position (sequence_query). Every answer of the query is found by implicit_mixture (from nstarts starts, by default 16 per possible completion of the $u$ unknown positions; kwargs go to it) and read per position by position_marginals. The same relation answers a continuation ([a, b, nothing]), a backward prediction ([nothing, b, c]) and an infilling ([a, nothing, c]).
LenticulumLM.window_relation — Method
window_relation(v::Vocabulary, P; s = 0.05, schedule = VPSDE(), nodes = nothing) -> (relation, ps, st)The closed-form implicit relation of an $n$-gram table: P is an $n$-dimensional array of size $V \times \cdots \times V$ whose entry P[a₁, …, aₙ] is the probability of the window $(a_1, \dots, a_n)$. The relation is a GaussianMixtureEps with one component of width s at the codes of every window with positive probability, weighted by it. It is the relation a diffusion model trained on these windows would converge to, with every derivative in closed form. nodes default to field_nodes with 8 antithetic pairs per level; the field is scaled by density_lambda, so its energy is a calibrated negative log-density.