Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.
cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).
DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JustVugg