serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.
cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).
DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ZacharyZcR