colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts
Engine (c/glm.c): MLA attention with compressed KV, sigmoid noaux_tc router, int8/int4/int2 quant kernels (AVX2), per-layer LRU expert cache + pinned hot-store, batch-union MoE, native MTP speculative decoding (lossless), multi-stop + official chat template, RAM auto-budget from MemAvailable. Tokenizer: byte-level BPE in C. Tooling: coli CLI, disk-safe FP8→int4 converter, tiny-random oracle validation (TF 32/32, greedy 20/20). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,18 @@
|
||||
"""Profila dove va il tempo: lettura expert dal disco vs attenzione vs moe vs matmul."""
|
||||
import cProfile, pstats, io, glob, json
|
||||
from engine import OlmoeStreaming
|
||||
|
||||
snap = glob.glob("/home/vincenzo/.cache/huggingface/hub/models--allenai--OLMoE-1B-7B-0924/snapshots/*")[0]
|
||||
ref = json.load(open("ref.json"))
|
||||
m = OlmoeStreaming(snap, expert_cap=16)
|
||||
|
||||
pr = cProfile.Profile()
|
||||
pr.enable()
|
||||
m.generate(ref["prompt_ids"], 8, greedy=True) # 8 token bastano per il profilo
|
||||
pr.disable()
|
||||
|
||||
s = io.StringIO()
|
||||
ps = pstats.Stats(pr, stream=s).sort_stats("tottime")
|
||||
ps.print_stats(15)
|
||||
print(s.getvalue())
|
||||
print(f"Hit-rate: {m.cache.hitrate()*100:.1f}% hit={m.cache.hits} miss={m.cache.miss}")
|
||||
Reference in New Issue
Block a user