Tokens as coordinates. A sequence of tokens over symbols is a point of : one block of coordinates per position, each token its scaled one-hot code. A query clamps the blocks of the known tokens and frees the rest, so the direction of a query (continue, predict backwards, fill in) is a choice of precisions, not of model.
Sources: code:
tokens.jl; Li et al., Diffusion-LM Improves Controllable Text Generation, NeurIPS 2022 (continuous codes for tokens); Austin et al., Structured Denoising Diffusion Models in Discrete State-Spaces, NeurIPS 2021 (the discrete alternative)Bibliography: Li et al. 2022 · Austin et al. 2021
Lenticulum (vault): Implicit Diffusion Learners §2 (polarity is a precision vector) · MNIST in Both Directions (a label as a scaled one-hot code)
1. The encoding
Vocabulary(symbols; scale = C) numbers the symbols and gives symbol the code
. Position of a sequence owns the coordinates block(v, i)
, so a sequence is the concatenation of its codes:
decode reads each block back by its largest coordinate, which is the nearest code: the codes
are orthogonal and of equal length, so the nearest code to is .
sequence_query(v, tokens) turns a partly known sequence (a symbol or nothing per position)
into the query of an implicit learner. The precision is on the blocks of known tokens
(hard clamps) and on the others (free), and holds the known codes. An unknown
position starts at zero, the centre of the codes, which favours no symbol.
This is the encoding of the MNIST example in Lenticulum, where the label is the code next to the image. A language model has only labels.
2. What it does not do
- It does not scale with the vocabulary. A block has coordinates, so a sequence of tokens over a vocabulary of is a state of coordinates, and every free block of a query is unknowns for the root finder. Diffusion-LM avoids this with learned low-dimensional embeddings, and masked discrete diffusion (D3PM, MDLM) avoids it by not having a continuous state at all. Both are open here (Open Problems in Assembling Language Models L3, L4).
- No tokeniser. Symbols are whatever the vocabulary is built from (characters, words, integers). Splitting text into tokens is left to the caller.
- No soft evidence. A query either clamps a position or frees it. A noisy observation of a
token would be a finite precision on its block, which the implicit learner supports but
sequence_querydoes not build yet. - The scale matters, and is not chosen here. Relative to the noise levels of the learner, sets how far apart the codes are: too small and the answers of neighbouring symbols merge under the smoothing, too large and the field is stiff. The default with mixture width separates them by 20 widths.
Related: sequence, Assembling a Language Model from Implicit Learners