LenticulumLM
A diffusion language model assembled from implicit learners. A language model is usually one network that runs one way. LenticulumLM builds one from parts: each part is a relation over a few token positions, the model is their product, and every query is answered by root finding with the known tokens clamped. Continuation, backward prediction and infilling are the same model asked with different clamps.
It builds on Lenticulum.jl, which provides the implicit learners (diffusion models read as relations), their products, and their beliefs.
Status: research prototype. Julia, not registered, one author. What is here is exact against closed forms; learned windows, a discrete learner and scale are open (roadmap).
The idea
Text of tokens over a vocabulary of symbols is a point of , one block of coordinates per position, each token its one-hot code. A relation on a window of positions is an implicit learner on coordinates. Windows at every offset, joined by separators on their overlaps,
are an -gram model when the windows are its -gram marginals, and a query clamps whatever is known:
| query | clamped | answered |
|---|---|---|
| continuation | the first tokens | the distribution of each later position |
| backward prediction | the last tokens | the distribution of each earlier position |
| infilling | both ends | the middle |
Every answer of a query is a stable root of the model’s field, with a Laplace weight, and the roots read per position are categorical beliefs.
A minimal example
using LenticulumLM
v = Vocabulary(['x', 'y'])
T = [0.9 0.1; 0.4 0.6] # a bigram chain, stationary π = (0.8, 0.2)
π₀ = [0.8, 0.2]
model, ps, st = sequence_relation(v, window_relation(v, π₀ .* T), window_relation(v, π₀), 3)
pos, P, _ = token_marginals(v, model, ps, st, ['x', nothing, 'y']) # fill in the middle
P # (0.6, 0.4) = T(x, ·) .* T(·, y), normalisedOn a three-symbol chain, continuation, backward prediction and infilling all agree with the
closed form to , with every completion found as its own answer
(src/sequence.md §3).
Layout
The repository mirrors Lenticulum.jl’s:
| where | what |
|---|---|
src/ | the package; an implementation note (.md) beside each .jl file |
test/ | the test suite, against closed forms |
vault/ | the Obsidian vault: design, roadmap, bibliography (Start Here) |
docs/ | Documenter pages, Literate tutorials, and the Quartz build of the vault (docs/site/) |
flake.nix | the development shells: every tool the experiments need |
experiments/ | one directory per experiment, each with its own environment (planned) |
Setting up
LenticulumLM is a workspace for experiments rather than a library to install. The Nix flake sets up everything: Julia (yours if it is on the PATH, else its own 1.12), the docs toolchain, the Hugging Face CLI, and on request Lean, Metamath or Isabelle (plan):
nix develop # Julia, Node, Python with the hf CLI, git-lfs
nix develop .#gpu # the same, with the host's NVIDIA driver visible to CUDA.jl
nix develop .#lean # + elan; also .#metamath, .#isabelle, .#proversData and checkpoints live outside the repository, in $LENTICULUMLM_DATA and
$LENTICULUMLM_MODELS (default ~/.cache/lenticulum/lm/). With direnv, .envrc enters the
default shell on cd.
Without Nix
Neither LenticulumLM nor Lenticulum.jl is registered. Project.toml takes Lenticulum’s
packages from GitHub, so a clone instantiates on its own:
using Pkg
Pkg.activate("path/to/LenticulumLM")
Pkg.instantiate()
Pkg.test()Developing against a local Lenticulum
To change both repositories at once, develop the local Lenticulum packages into the environment. This overrides the GitHub sources in the (untracked) manifest:
Pkg.develop([PackageSpec(path = "../Lenticulum.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/LenticulumCore.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/Mycelium.jl"),
PackageSpec(path = "../Lenticulum.jl/lib/VariationalDiffusion.jl")])Reading the theory
- The vault, on the web at mathstruct.org/LenticulumLM (the API
documentation is under api/dev), or in Obsidian:
open the repository root and start at
vault/Start Here.md. - Lenticulum’s vault, for the implicit learners themselves: mathstruct.org/Lenticulum.jl.
- The CT-ML wiki, for the general category theory both build on: mathstruct.org/CategoryTheory-ML-Wiki.
License
MIT, see LICENSE.