Tokens for every profile, and the parse limits. Source:
lexer.rs.code:
lexer.rs
Tokens are:
- identifiers, and backquoted operator names;
- literals:
0x…is an unsigned 64-bit integer; floats use Rust’s shortest round-trip form; strings and characters take escapes; #hex(a content id) and##hex(a nominal id);@markers;- symbols, including both
-->and→; - newlines, which only blocks treat as significant.
# followed by anything other than 64 hex digits starts a comment, as does //.
Limits default to 512 KiB, 2²⁰ expression nodes and 4096 definitions per text (SR-50). They bound size, never nesting (2026-10-24, SP-015 / SR-145): the depth limit of 128 is gone; depth costs memory (explicit stacks), never a refusal or a crash.
Tok implements Display for diagnostics: a token as written, in backquotes (`)`, `@prim`, `##023be898216c…` shortened), a string or character literal quoted, and “end of line” or “end of text”.