measurement julia

Which facts about Julia code Sophia knows, which it approximates, and which it leaves open, measured against Julia itself. The conformance suite the external review asked for (SR-129), started 2026-10-20 with its first two parts: generic-function ownership on a fixture and on LinearAlgebra.

Sources: crates/sophia-ingest/tests/generics.rs and tests/julia-semantics/generics/ (the fixture); tests/julia-semantics/methods.jl (Julia’s ground truth, methods); examples/julia_conformance.rs (julia_conformance); Julia Name Resolution.

Principle

For each Julia construct, compare what Julia actually does (by running Julia and asking it) with what Sophia records. Each comparison yields one of:

  • knows: Sophia’s fact equals Julia’s;
  • approximates: Sophia’s fact differs in a known, bounded way, and the page says how;
  • leaves open: Sophia records “unknown” (a refusal, a dynamic include, an unplaced method);
  • wrong: Sophia states something false. Every “wrong” is a bug to fix or report.

The suite is not a Julia implementation. It is a ledger of which claims Sophia may make.

Part 1: generic ownership on a fixture (in CI)

tests/generics.rs on the package Gen, src/Gen.jl and src/sub.jl:

CaseJulia’s meaningSophia
f in module A and in module Btwo functions, Gen.A.f, Gen.B.fjl.Gen.A.f (4 methods), jl.Gen.B.f (1)knows
f in Gen itselfa third function, Gen.fjl.Gen.f (2)knows
A.f(p::Point) = … in Gena method of Gen.A.fmember of jl.Gen.A.fknows
import ..A: f then f(s::String) = … in Gen.Ca method of Gen.A.fmember of jl.Gen.A.fknows
import Base: show; show(io, p) = …a method of Base.showmember of jl.Base.show; no jl.Gen.showknows
import Base.length; length(p) = …a method of Base.lengthmember of jl.Base.lengthknows
Base.:+(p, q) = …a method of Base.:+member of jl.Base.`+` knows
Point() = …, Point(a, b) = …constructors of Gen.Pointjl.Gen.Point.new (2)knows
use(p) = f(p) + f(p, p)calls Gen.fnominal_refs = [jl.Gen.f]knows

Part 2: generic ownership on LinearAlgebra (Julia 1.13.1)

Julia lists 4 933 methods from LinearAlgebra’s source files. Sophia stores 2 920 methods (normaliser v7; on v6 it was 2 939, see the check row).

OutcomeMethodsWhat
same owner2 777 (95.1 %)knows
different owner: re-exported binding86approximates: Base.Matrix is Core.Array, Base.cbrt is Base.Math.cbrt, Base.uppercase is Base.Unicode.uppercase, Base.CartesianIndex is Base.IteratorsMD.CartesianIndex (SP-013 Julia bindings of modules Sophia has not ingested)
different owner: spurious unit in a macro call0 (11 on v6)was wrong: @deprecate f(…; check::Bool = true) … gave a “method” named check. Reported to Moonkale, fixed in v7 the same day (nothing inside a call is a module-level unit)
no Julia method at that line57approximates: @static if Sys.isapple() code (not compiled on Linux), function f end (no method), multi-line signatures
Julia methods at lines where Sophia stores none1 059leaves open: @eval loops, macros, default constructors, refused definitions

Before the (module, name) fix (--by-name reproduces the old grouping): 1 578 of 2 939 (53.7 %) on v6, 1 578 of 2 920 (54.0 %) on v7. The 1 296 disagreements (v6) were 898 methods of Base functions filed as LinearAlgebra’s (662 by import, 236 by qualified heads such as Base.to_indices, stored as a function named `Base.to_indices`), 234 methods of BLAS and LAPACK functions filed under LinearAlgebra, 38 of Base.Broadcast, and the alias cases that remain.

Part 3 onwards: planned

Each part is one fixture package (or a real one) plus a Julia script that prints the ground truth.

PartConstructJulia’s ground truthSophia’s claim
3dispatch: which method a call reacheswhich(f, Tuple{…}) for chosen signaturesnone yet; record Julia’s answer as a dispatch fact (Julia Multiple Dispatch and World Age)
4world change: adding a method or an importmethods(f) before and after; Base.get_world_counter()which facts become stale (SR-85)
5macrosmacroexpand, Meta.lowerexpansions as transformation records (Julia Macros and Generated Functions)
6closures and captured variablesfieldnames(typeof(closure))references inside a unit (v4 scoping)
7mutable globals (const vs non-const, global x)binding partition / world semantics in Julia ≥ 1.12whether a global reference is content or state
8parser agreementJuliaSyntax.jl’s tree vs tree-sitter’srefusals and silent disagreements
9include orderpathof, file/line of each definitioninclude graph (module::discover)

How to run

cargo test -p sophia-ingest --test generics
julia crates/sophia-ingest/tests/julia-semantics/methods.jl LinearAlgebra > truth.tsv
cargo run --release -p sophia-ingest --example julia_conformance -- <stdlib>/LinearAlgebra truth.tsv