New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA
walker) wired into spec_decode as a third draft source ("metodo F"), tried
before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the
forced span is tokenized and injected as drafts; the existing batch-union
verification confirms them, so a wrong or out-of-sync grammar can never
change the output. Lazy arming skips preambles; adaptive guard (same
pattern as MTP) disables the source below 50% acceptance; grammar-accepted
tokens no longer pollute the MTP acceptance counter.
GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and
with the int4 MTP head from #8); GRAMMAR_DRAFT=n caps the span (default 24).
Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON
classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok),
100% acceptance (48/48), output byte-identical to baseline.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.
cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).
DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sidd