colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts

Engine (c/glm.c): MLA attention with compressed KV, sigmoid noaux_tc router,
int8/int4/int2 quant kernels (AVX2), per-layer LRU expert cache + pinned
hot-store, batch-union MoE, native MTP speculative decoding (lossless),
multi-stop + official chat template, RAM auto-budget from MemAvailable.
Tokenizer: byte-level BPE in C. Tooling: coli CLI, disk-safe FP8→int4
converter, tiny-random oracle validation (TF 32/32, greedy 20/20).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
JustVugg
2026-07-05 20:52:05 +02:00
parent a6deef8e44
commit 1ae22a6135
32 changed files with 4440 additions and 1 deletions
+20
View File
@@ -0,0 +1,20 @@
"""Sweep della cache: per ogni capacita' misura correttezza, hit-rate, RAM cache, velocita'."""
import json, time, glob
from engine import OlmoeStreaming
snap = glob.glob("/home/vincenzo/.cache/huggingface/hub/models--allenai--OLMoE-1B-7B-0924/snapshots/*")[0]
ref = json.load(open("ref.json"))
exp = ref["full_ids"][len(ref["prompt_ids"]):]
n_new = len(exp)
EXPERT_MB_BF16 = 12.6
print(f"{'cap':>4} {'RAMcache':>9} {'match':>6} {'hit%':>6} {'tok/s':>7} {'sec':>6}")
for cap in (16, 32, 48, 64):
m = OlmoeStreaming(snap, expert_cap=cap)
t = time.time()
out = m.generate(ref["prompt_ids"], n_new, greedy=True)
dt = time.time() - t
gen = out[len(ref["prompt_ids"]):]
match = sum(a == b for a, b in zip(gen, exp))
ram = cap * m.L * EXPERT_MB_BF16 / 1024
print(f"{cap:>4} {ram:>7.1f}GB {match:>3}/{n_new:<2} {m.cache.hitrate()*100:>5.1f}% {n_new/dt:>7.2f} {dt:>6.1f}")