implementation measurement

What Sophia physically stores today (80f71ee and after), table by table, classified by Object Categories: authoritative, redundant (an index), derivable (a materialised view), or cache. Measured 2026-10-20 on SQLite. Part of Storage and Materialization.

Sources: crates/sophia-store/src/sqlite.rs (schema), kv.rs (the key-value layout of redb and LMDB); SQLite’s dbstat virtual table; corpora as in Real Code Corpora.

The tables

SQLite table (KV table)HoldsCategoryClassification
t_definition (Rec)per record: id, language, normaliser, kind, component, index, size, canonical bytes, display, status, refs, nominal refs2 (and 1 for sophialang)id, kind, status: needed. Canonical bytes: derivable for ingested code. Display: derivable (labels, variable names: from source). refs blob: redundant with t_ref
t_definition_component, _kind (Comp, Kind)indexes5redundant (indexes)
t_ref + t_ref_dst (RefRev)reference edges, both directions2 / 5forward: needed by queries (or derivable from the refs blob); reverse: index
t_nominal_refuses of nominal entities (generic calls)2needed; derivable for ingested code
t_nominal, t_relation (+ _member)nominal entities and their members (method tables, library roles)3authoritative for authored code; derivable for an ingested library (it is the ingest’s grouping)
t_name (+ _target)names3authoritative (for ingested code: derivable)
t_lineage (+ _id)lineages3authoritative
t_annotation (+ _subject)annotations (proof links, roles)3 / 4authoritative when authored; for Metamath derivable
t_fact (+ _subject)derived facts with provenance4 / 5per fact
t_logthe change log3authoritative (the history)
t_metasettings, rendering profiles, cached query results (query-result/<id>), saved queries3 / 5profiles and saved queries authoritative; query results a cache with no eviction (SP-014 The query result cache is never evicted)

Source text of ingested files was not stored before 2026-10-20 (see below). For ingested code it is the authoritative object, and everything Sophia stores about it is derived from it.

Measured: Julia LinearAlgebra (module mode, v6)

Source: 34 files, 1.34 MB (0.23 MB with zstd -9). Store: 17.7 MB SQLite, 3 288 records.

PartKiBShareClass
canonical bytes (in t_definition)12 03670 %derivable from source (WP-66 parts, about 9× the source)
rest of t_definition (ids, kinds, display 25, refs 63)≈ 2 67015 %ids needed; rest derivable or redundant
t_log9005 %history of the ingest itself
method tables (t_relation + index)4683 %derivable (ingest grouping)
reference edges (t_ref + t_ref_dst)3242 %needed / index
indexes (component, kind, autoindex)3762 %redundant
names, nominal, nominal refs, meta, lineage≈ 4903 %mixed

Measured: Metamath set.mm

Source: 51.7 MB (17.0 MB with zstd -9). Store: 567 MiB SQLite, 98 136 records (48 618 statements, 48 035 proofs, 1 483 syntax).

PartMiBShareClass
proof canonical bytes8214 %derivable from source (the proofs are authored in the .mm)
statement canonical bytes499 %derivable from source
refs blobs on records468 %redundant with t_ref
rest of t_definition (ids, components, display, kinds)≈ 6111 %ids needed; rest derivable
t_ref12422 %needed (or derivable from the blobs)
t_ref_dst11520 %index
t_log275 %ingest history
names, lineage, annotations, relations, indexes≈ 6311 %derivable for an ingested library

What this says

  1. For ingested code, the whole store is a materialised view of the source. set.mm: 567 MiB derived from 17 MB of compressed authority. A store that kept only the source blobs, the ids, the kinds and one copy of the edges would hold everything a query needs and could rebuild the rest. Rough estimate for set.mm: 17 MB of source, 98 136 × 32 B ids ≈ 3 MB, 1.5 M edges as 8-byte surrogate pairs (SP-011 option 1) ≈ 24 MB and a reverse index of the same size: under 70 MB, against 567 MiB. To be measured, not assumed (Storage Measurements M-1, M-2).
  2. References are stored three times (blob, forward table, reverse index). One copy plus an index is enough (SP-011 options 2 and 4).
  3. The canonical form of WP-66 is large. 9× the source for Julia. It is fine as a hashing input and wasteful as storage. If it is kept at all, it should be compressed or rebuilt.
  4. The log records the ingest itself. For a deterministic ingest (source, frontend version) one log entry naming the source blob and the frontend would replace thousands of entries: the ingest is a transformation record (SPEC-006 §6a), not a series of edits.
  5. Query results accumulate forever in t_meta (SP-014 The query result cache is never evicted).
  6. Not all of this is waste. Rebuilding the canonical bytes of one definition requires parsing its file. If Moonkale shows a definition’s canonical form often, keeping it (compressed) may be right. That is a measured decision, per Materialization Policy.

Sources and manifests, and reconstruction (2026-10-20)

The external review (Review 2026-10-20 Real Code 2 (external)) made the precondition explicit: before any derived bytes are dropped, the store must hold an immutable source and everything needed to rebuild and verify each record from it. Now it does, for module-mode Julia and for Metamath (source):

  • each source file is a content-addressed record (language source);
  • an ingest manifest records the frontend and its version, the normaliser, the package, and per file its path, source id and include module, in reading order;
  • sophia reconstruct NAME rebuilds every record from the store alone and compares it field by field.
CorpusSourcesRecords rebuiltIdenticalTimeStore growth
Julia LinearAlgebra34 files, 1.34 MB3 2843 2842.0 s17.7 → 19.1 MB
Metamath set.mm1 file, 51.7 MB98 13698 13617.5 s567 → 617 MiB

Tested on all four backends (crates/sophia-ingest/tests/reconstruct.rs), including that a changed source is detected and that every stored Julia record is among the rebuilt ones. Sources are stored uncompressed for now; compressing them is Compression item 4.

This makes items 1 and 2 of Compression safe to try, as measurements on a copy, each followed by reconstruct. Nothing has been dropped yet.

What is not changed yet

Nothing in the storage layout has been changed on the strength of this page (SR-126). The first change, when Daniel agrees, would be the cheapest and least risky one: drop the refs blob where an edge table exists (SP-011 option 2), then measure the rest.