LOOKA: routing predictability counters; PILOT: router-lookahead disk prefetch (async I/O thread); cap auto-raise to RAM budget (#12)
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable one full layer ahead (79.4% skipping attention only; 41.3% previous-token). PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated disk queue (+92s/48 tok, measured), hence the lock-free ring + worker. Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off. cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts, CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8 (#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same). DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same 48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT re-evaluation belongs to warm-cache serve sessions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -91,7 +91,7 @@ The engine at runtime is pure C — python is only used by the one-time converte
|
||||
|
||||
### Experimental resident CUDA backend
|
||||
|
||||
This fork includes an opt-in CUDA backend for model-resident tensors. Streaming
|
||||
colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming
|
||||
experts deliberately remain on the original CPU path for now: copying an expert
|
||||
from NVMe to the GPU on every use would only replace the disk bottleneck with a
|
||||
PCIe bottleneck. Resident quantized tensors are uploaded lazily once and reused.
|
||||
@@ -153,7 +153,11 @@ The fixture has random weights and is not a language model. It exists only to
|
||||
preserve the real MLA/MoE/streaming shapes and compare CPU streaming, dense-only
|
||||
CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.
|
||||
|
||||
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `TF=1` teacher-forcing validation.
|
||||
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache.
|
||||
|
||||
**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.**
|
||||
|
||||
**Router-lookahead prefetch** (`PILOT=1`, experimental): GLM-5.2's expert routing is measurably predictable *ahead of time* — applying layer L+1's router to layer L's post-attention state recalls **71.6%** of the true top-8 (vs 41.3% for "same experts as last token"). `PILOT=1` uses this to issue next-layer expert readahead from a dedicated I/O thread while the current layer computes. On our dev box the disk is already ~80% saturated, so it measures neutral; on machines where compute and disk are balanced (like the Ryzen AI 9 in issue #12: 43% disk / 46% matmul) it should overlap real work — measurements welcome.
|
||||
|
||||
**The learning cache**: the engine records which experts your usage actually routes to (`.coli_usage` next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user