implementation reference

The benchmark suite (SR-72) and how its results are recorded. Every hypothesis in the design notes that depends on performance has a benchmark here, and every benchmark has a hypothesis. Results go to Benchmark Results, with the machine, the commit, the date and the exact command. A result that contradicts a design note is recorded as such, and the note is corrected (SR-73).

Sources: criterion.rs; Mytkowicz et al., Producing Wrong Data Without Doing Anything Obviously Wrong!, ASPLOS 2009 (measurement bias); Georges, Buytaert and Eeckhout, Statistically Rigorous Java Performance Evaluation, OOPSLA 2007. code: crates/sophia-bench (benches in benches/, generators in src/, the report writer in src/bin/sophia-bench-report.rs).

Principles

  • Reproducible inputs. Synthetic stores are generated from a seed with realistic distributions: definition sizes log-normal, references per definition Poisson, recursive groups rare (about 1 %), name depth 2–5. Real corpora (sophialang samples, and later ingested Julia and Rust) are pinned by commit.
  • Statistics, not anecdotes. criterion’s confidence intervals; at least 30 samples; outliers reported, not hidden; machine noise controlled (performance governor, no other heavy work), and the conditions written down.
  • The same suite on every backend. Backend benchmarks are generic over the Backend trait.
  • Sizes from 10³ to 10⁶ definitions for store benchmarks. 10⁷ when a backend gets that far.
  • Real code, not only synthetic (SR-83): see B-16.
  • Distributions and memory: latencies are reported as p50, p90 and p99 with cold and warm caches, together with allocations and resident memory, not only medians.
  • Scripted. cargo bench -p sophia-bench runs everything. cargo run -p sophia-bench --bin sophia-bench-report writes a Markdown table that is pasted into Benchmark Results.

The benchmarks

#WhatHypothesis testedFrom
B-01BLAKE3 throughput by input size (32 B – 64 KiB); our framing’s overheadH2: hashing a definition is < 1 µsBlock Size and Hash Overhead
B-02normalise + encode + hash a definition: flat layout vs boxed treeH5: flat ≥ 3× fasterBlock Size §3
B-03bulk ingest of N definitions (put component, names, log) per backendwhich backend ingests fastest; scaling with NDatabase Comparison
B-04point lookup by id and by name; dependants and dependencies (one hop); transitive closure from a rootread latency per backend; closure costDatabase Comparison
B-05store size on disk per backend, with digests vs local ids for referencesH3: local ids save ≥ 30 %Block Size §2
B-06storage encodings: canonical TLV vs compact varint, with conversion costH6Block Size §2
B-07corpus statistics: definition sizes, references, dedup ratio by subterm threshold TH1, H4Block Size §5, Compression and Canonical Form
B-08zstd with and without a trained dictionary on stored bytesthe size and latency trade-offCompression §4
B-09invalidation precision: facts made stale by adding a method, against the facts actually affectedSP-006’s proposal is precise enoughSP-006 Open method tables versus content addressing
B-10print and parse latency per profile; a round trip per definitioninteractive editing is < 5 ms per definitionSPEC-005 Rendering Profiles
B-11query latency: SQL views vs Datalog vs the backend’s own dialect—SPEC-001 §11
B-12interpreter speed on small kernels (fib, sum, nbody)—M4
B-13compiled code against C, Rust and Julia on the same kernelsSR-52: within 1.2×M5
B-14GC pause times and throughput; marking on a separate coreSR-56M7
B-15edit-to-updated-graph latency through sophia-interface: fetch, edit, parse, normalise, store, the change feed, and Moonkale’s refresh (measured on Moonkale’s side). Cold and warm; p50/p90/p99Moonkale stays interactive: < 50 ms p99 for one definition on 10⁵ storesreview §7

B-15 measured (2026-10-20, late night): p99 of the whole loop 2.4–3.5 ms on 10⁵ definitions, target < 50 ms; the change feed’s 500 ms polling dominates what a user would see (Benchmark Results). The graph-panel and paint parts below are still open.

B-15 measurement plan (agreed with the Moonkale agent, 2026-10-20). Moonkale’s graph panel refreshes on a graph_epoch bump after its own saves, not through the change feed. So the measured path is: apply() returns → the epoch bump → the renderer payload is delivered, timed with tracing timestamps, p50/p99 over about 200 saves. Paint time on the agent’s headless sway with Mesa software GL is CPU-bound and is reported separately, marked “software GL, not hardware”. The real paint number comes from Daniel’s desktop. | B-16 | real corpora: a substantial Julia package, a Rust workspace, a theorem library (Metamath set.mm), and a code base with many tiny definitions. These need frontends or Moonkale’s WP-66 normal forms | H1, H3, H4 on real code | review §7 | | B-17 | query workloads in Sophia’s query language (SPEC-007) on every backend: direct and transitive dependants, transitive dependencies, count by kind, unreferenced definitions, names by prefix; native executor with and without magic sets; on SQLite also the SQL translation. Plus the zero-copy breakdown of an LMDB record read (SR-101) | S-ADR-0010’s re-decision; SR-96 (backend independence costs little); magic sets matter | Daniel’s query proposal |