danielkrause/colibri-strix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

Browse Source 
* feat: standardize runtime output in English

* test: cover English CLI output
This commit is contained in:
Sidd
2026-07-12 17:08:40 +05:30
committed by GitHub
parent 6e7aa6f92e
commit 2416bc9079
24 changed files with 330 additions and 280 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -4
View File
@@ -1,5 +1,5 @@
<p align="center">
<img src="assets/colibri.svg" width="500" alt="colibrì — piccolo motore, modello immenso">
<img src="assets/colibri.svg" width="500" alt="colibrì — tiny engine, immense model">
</p>
**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
@@ -7,7 +7,7 @@
```
$ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ pronto in 32s · residente 9.9 GB
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
```
@@ -154,7 +154,7 @@ make test-python # run Python tests (requires python)
# Verify (tiny model, 2.4 MB):
pip install torch transformers safetensors huggingface_hub
python tools/make_glm_oracle.py # generate tiny oracle
SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 posizioni"
SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions"
# Run with real model:
SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference
@@ -313,7 +313,7 @@ works against the colibrì OpenAI-compatible server (in review, #21) or any othe
compatible endpoint. Nothing leaves the endpoint you configure. The terminal
`coli chat` remains the first-class interface.
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache.
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache.
**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.**
assets/colibri.svg
+2 -2
View File
@@ -49,7 +49,7 @@
<text x="252" y="62" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="52" font-weight="bold" fill="#00afaf">colibr&#236;</text>
<text x="252" y="94" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="19" fill="#808080" font-style="italic">piccolo motore, modello immenso</text>
font-size="19" fill="#808080" font-style="italic">tiny engine, immense model</text>
<text x="252" y="122" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="15" fill="#9a9a9a">GLM-5.2 &#183; 744B MoE &#183; int4 &#183; streaming CPU</text>
</svg>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 3.5 KiB

c/Makefile
+1 -1
View File
@@ -14,7 +14,7 @@ ifneq ($(OMPDIR),)
OMPC = -Xclang -fopenmp -I$(OMPDIR)/include
OMPL = -L$(OMPDIR)/lib -lomp
else
$(warning libomp non trovato: build SINGLE-THREAD. Per il multithread: brew install libomp)
$(warning libomp not found: building single-threaded. For multithreading: brew install libomp)
OMPC =
OMPL =
endif
c/coli
+59 -59
View File
@@ -1,24 +1,24 @@
#!/usr/bin/env python3
"""
colibrì — piccolo motore, modello immenso.
CLI per far girare GLM-5.2 (744B) in locale, su CPU, in ~15-26 GB di RAM.
colibrì — tiny engine, immense model.
Run GLM-5.2 (744B) locally on CPU with roughly 15-26 GB of RAM.
coli chat chat interattiva (carica il modello UNA volta)
coli serve API HTTP compatibile OpenAI (motore persistente)
coli run "prompt" generazione singola
coli info stato: modello, RAM, disco, config
coli plan piano risorse Disk / RAM / VRAM
coli doctor diagnosi installazione e piano di esecuzione
coli bench [task...] benchmark di qualità (MMLU/HellaSwag/...)
coli convert converte GLM-5.2-FP8 -> int4 (streaming)
coli build compila il motore
coli chat interactive chat (loads the model once)
coli serve OpenAI-compatible HTTP API (persistent engine)
coli run "prompt" one-shot generation
coli info model, RAM, disk, and configuration status
coli plan Disk / RAM / VRAM resource plan
coli doctor installation and execution-plan diagnostics
coli bench [task...] quality benchmarks (MMLU/HellaSwag/...)
coli convert convert GLM-5.2-FP8 to int4, one shard at a time
coli build build the engine
Config via env o flag (validi anche dopo il sottocomando):
COLI_MODEL=<dir> modello (default /home/vincenzo/glm52_i4)
--ram N budget RAM in GB (auto-cap cache expert)
--repin N adatta gli expert RAM/VRAM ogni N token
--topp P top-p adattivo sugli expert --topk N top-k fisso
--ngen N token massimi per risposta --cap N slot cache/layer
Configuration through environment variables or flags (also valid after the subcommand):
COLI_MODEL=<dir> model directory (default /home/vincenzo/glm52_i4)
--ram N RAM budget in GB (automatically sizes the expert cache)
--repin N adapt RAM/VRAM experts every N tokens
--topp P adaptive expert top-p --topk N fixed top-k
--ngen N maximum response tokens --cap N cache slots/layer
"""
import os, sys, subprocess, argparse, json, time, signal, shutil, threading, re, codecs, tempfile, textwrap
@@ -82,7 +82,7 @@ def banner(sub=""):
sp=sprite_lines()
txt=[
f"{C.teal}{C.b}colibrì{C.r} {C.dim}v1.0{C.r}",
f"{C.dim}piccolo motore, modello immenso{C.r}",
f"{C.dim}tiny engine, immense model{C.r}",
f"{C.gray}GLM-5.2 · 744B MoE · int4 · streaming CPU{C.r}",
f"{C.dgray}{sub}{C.r}" if sub else "",
"",
@@ -100,11 +100,11 @@ def term_w(): return min(shutil.get_terminal_size((80,20)).columns, 100)
def need_model(model):
if not os.path.isdir(model):
sys.exit(f"{C.yel}modello non trovato:{C.r} {model}\n imposta COLI_MODEL o usa --model")
sys.exit(f"{C.yel}model not found:{C.r} {model}\n set COLI_MODEL or use --model")
if not os.path.exists(os.path.join(model,"tokenizer.json")):
sys.exit(f"{C.yel}manca tokenizer.json in {model}{C.r}")
sys.exit(f"{C.yel}tokenizer.json is missing from {model}{C.r}")
if not os.path.exists(GLM):
sys.exit(f"{C.yel}motore non compilato.{C.r} Esegui: coli build")
sys.exit(f"{C.yel}engine is not built.{C.r} Run: coli build")
def cuda_binary():
if not os.path.exists(GLM) or sys.platform != "linux": return False
@@ -149,7 +149,7 @@ def env_for(a):
ram,ctx,devices,vram=resource_request(a,e)
plan=build_plan(a.model,ram,ctx,devices,vram)
except (OSError,ValueError,json.JSONDecodeError) as error:
sys.exit(f"{C.yel}piano risorse non valido:{C.r} {error}")
sys.exit(f"{C.yel}invalid resource plan:{C.r} {error}")
has_cuda=cuda_binary()
e=environment_for_plan(plan,e,has_cuda)
rt=plan["tiers"]["ram"]; vt=plan["tiers"]["vram"]
@@ -292,26 +292,26 @@ def cmd_info(a):
def row(k,v): print(f" {C.gray}{k:<10}{C.r} {v}")
if os.path.exists(cfgp):
c=json.load(open(cfgp))
row("modello", a.model)
row("model", a.model)
row("arch", f"hidden {c.get('hidden_size')} · {c.get('num_hidden_layers')} layer · "
f"{c.get('n_routed_experts')} expert/layer · top-{c.get('num_experts_per_tok')}")
sts=[x for x in os.listdir(a.model) if x.endswith('.safetensors')]
sz=sum(os.path.getsize(os.path.join(a.model,x)) for x in sts)
row("shard", f"{len(sts)} file · {sz/1e9:.0f} GB su disco")
row("shards", f"{len(sts)} files · {sz/1e9:.0f} GB on disk")
else:
print(f" {C.yel}config.json non presente (conversione incompleta?){C.r}")
print(f" {C.yel}config.json is missing (incomplete conversion?){C.r}")
try:
mi=open('/proc/meminfo').read()
tot=int(re.search(r'MemTotal:\s+(\d+)',mi).group(1))/1e6
av=int(re.search(r'MemAvailable:\s+(\d+)',mi).group(1))/1e6
row("RAM", f"{tot:.0f} GB totali · {av:.1f} GB disponibili")
row("RAM", f"{tot:.0f} GB total · {av:.1f} GB available")
except Exception: pass
try:
fs = shutil.disk_usage(a.model if os.path.isdir(a.model) else HERE)
row("disco", f"{fs.free/1e9:.0f} GB liberi")
row("disk", f"{fs.free/1e9:.0f} GB free")
except OSError:
row("disco", "? GB (non disponibile)")
row("motore", "pronto ✓" if os.path.exists(GLM) else "da compilare (coli build)")
row("disk", "? GB (unavailable)")
row("engine", "ready ✓" if os.path.exists(GLM) else "not built (coli build)")
knobs=[]
if a.ram: knobs.append(f"ram {a.ram}GB")
if a.topp: knobs.append(f"topp {a.topp}")
@@ -323,11 +323,11 @@ def cmd_plan(a):
from resource_plan import build_plan, format_plan
try:
ram,ctx,devices,vram=resource_request(a,os.environ)
if ctx<1: raise ValueError("--ctx deve essere positivo")
if a.vram<0: raise ValueError("--vram non puo essere negativo")
if ctx<1: raise ValueError("--ctx must be positive")
if a.vram<0: raise ValueError("--vram cannot be negative")
plan=build_plan(a.model,ram,ctx,devices,vram)
except (OSError, ValueError, json.JSONDecodeError) as error:
sys.exit(f"{C.yel}impossibile creare il piano:{C.r} {error}")
sys.exit(f"{C.yel}cannot create resource plan:{C.r} {error}")
if a.json:
print(json.dumps(plan,indent=2))
return
@@ -339,9 +339,9 @@ def cmd_doctor(a):
from doctor import exit_code, format_doctor, run_doctor
try:
ram,ctx,devices,vram=resource_request(a,os.environ)
if ctx<1: raise ValueError("--ctx deve essere positivo")
if ram<0: raise ValueError("--ram non puo essere negativo")
if vram<0: raise ValueError("--vram non puo essere negativo")
if ctx<1: raise ValueError("--ctx must be positive")
if ram<0: raise ValueError("--ram cannot be negative")
if vram<0: raise ValueError("--vram cannot be negative")
except ValueError as error:
report={"schema_version":1,"status":"error","model":os.path.abspath(a.model),
"checks":[{"id":"config.arguments","status":"fail","summary":str(error)}],
@@ -354,7 +354,7 @@ def cmd_doctor(a):
def cmd_run(a):
need_model(a.model)
prompt=" ".join(a.prompt) if a.prompt else sys.exit('uso: coli run "il tuo prompt"')
prompt=" ".join(a.prompt) if a.prompt else sys.exit('usage: coli run "your prompt"')
banner("run")
# template ufficiale GLM-5.2: niente \n dopo i ruoli; <think></think> = risposta diretta (nothink)
e=env_for(a); e["PROMPT"]=f"[gMASK]<sop><|user|>{prompt}<|assistant|><think></think>"
@@ -367,24 +367,24 @@ def cmd_chat(a):
e=env_for(a); e["SERVE"]="1"
p=subprocess.Popen([GLM,str(a.cap)], env=e, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=errlog, bufsize=0)
sp=Spinner("sveglio il gigante (744B)…"); sp.start()
sp=Spinner("waking the giant (744B)…"); sp.start()
st=stream_turn(p, READY, lambda b: None)
sp.stop()
if st is None:
errlog.seek(0); print(errlog.read()[-1500:]); sys.exit("il motore è uscito durante il load")
errlog.seek(0); print(errlog.read()[-1500:]); sys.exit("the engine exited while loading")
errlog.flush()
try:
elog=open(errlog.name).read()
mload=re.search(r"caricato in ([0-9.]+)s \| densa residente: ([0-9.]+) MB", elog)
if mload: print(f" {C.grn}✓{C.r} pronto in {mload.group(1)}s {C.dim}· residente {float(mload.group(2))/1000:.1f} GB · RSS {st.get('rss','?')} GB{C.r}")
mload=re.search(r"loaded in ([0-9.]+)s \| resident dense: ([0-9.]+) MB", elog)
if mload: print(f" {C.grn}✓{C.r} ready in {mload.group(1)}s {C.dim}· resident {float(mload.group(2))/1000:.1f} GB · RSS {st.get('rss','?')} GB{C.r}")
for l in elog.splitlines(): # una riga di stato per riga, senza path
if l.startswith(("[RAM_GB","[PIN]","[MTP]","[USAGE]","[DSA]","[KV]")):
l=re.sub(r" ?\(?/[^ )]+\)?","",l.strip()) # via i percorsi lunghi
l=re.sub(r" da$| in$","",l)
l=re.sub(r" from$","",l)
for chunk in textwrap.wrap(l, term_w()-4) or [l]:
print(f" {C.dgray}{chunk}{C.r}")
except Exception: pass
print(f" {C.dim}scrivi e premi invio · :piu continua risposta · :reset memoria · :q esci{C.r}\n")
print(f" {C.dim}type and press Enter · :more continues · :reset clears memory · :q exits{C.r}\n")
w=term_w()-4
def user_box(msg):
"""ri-disegna il messaggio dentro una box che si ADATTA su piu' righe:
@@ -415,7 +415,7 @@ def cmd_chat(a):
if msg==":reset":
p.stdin.write(b"\x02RESET\n"); p.stdin.flush()
stream_turn(p, END, lambda b: None)
print(f" {C.dim}✦ memoria azzerata{C.r}\n"); continue
print(f" {C.dim}✦ memory cleared{C.r}\n"); continue
if msg in (":piu",":più",":more",":continua"):
p.stdin.write(b"\x02MORE\n"); p.stdin.flush()
else:
@@ -430,7 +430,7 @@ def cmd_chat(a):
pl=[l for l in tail.splitlines() if l.startswith("[prefill]")]
return pl[-1].replace("[prefill] ","prefill ") if pl else ""
except Exception: return ""
sp2=Spinner("pensa…", tick=prefill_tick); sp2.start()
sp2=Spinner("thinking…", tick=prefill_tick); sp2.start()
md=MDStream(" ") # markdown -> terminale, in streaming
raw=os.environ.get("COLI_RAW")=="1"
def echo(bs, _dec=dec, _st=state):
@@ -445,23 +445,23 @@ def cmd_chat(a):
st=stream_turn(p, END, echo)
if not raw: md.close()
sp2.stop()
if st is None: print(f"\n {C.yel}[motore terminato]{C.r}"); break
if st is None: print(f"\n {C.yel}[engine terminated]{C.r}"); break
el=time.time()-t0
if st.get("tok"):
print(f"\r {C.dgray}└─ {st['tok']} tok · {st['tps']:.2f} tok/s · hit {st['hit']:.0f}% · RSS {st['rss']:.1f} GB · {el:.0f}s{C.r}")
if st["tok"]>=a.ngen:
print(f" {C.yel}…troncato al limite --ngen ({a.ngen}): scrivi :piu per far continuare la risposta{C.r}")
print(f" {C.yel}…stopped at --ngen ({a.ngen}); type :more to continue the response{C.r}")
print()
else:
print()
except KeyboardInterrupt:
print(f"\n {C.dim}interrotto{C.r}")
print(f"\n {C.dim}interrupted{C.r}")
finally:
try: p.stdin.close(); p.terminate()
except Exception: pass
try: os.unlink(errlog.name)
except Exception: pass
print(f" {C.teal}ciao{C.r} {C.dim}— il colibrì torna al nido{C.r} 🐦\n")
print(f" {C.teal}goodbye{C.r} {C.dim}— the hummingbird returns to its nest{C.r} 🐦\n")
def cmd_serve(a):
need_model(a.model)
@@ -480,14 +480,14 @@ def cmd_bench(a):
# dataset mancanti -> li scarica una volta (fetch_benchmarks.py li mette in --data come JSONL)
missing=[t for t in tasks.split(",") if not os.path.exists(os.path.join(a.data,f"{t}.jsonl"))]
if missing:
print(f" {C.dim}scarico i dataset mancanti: {', '.join(missing)}{C.r}")
print(f" {C.dim}downloading missing datasets: {', '.join(missing)}{C.r}")
subprocess.call([py, os.path.join(TOOLS,"fetch_benchmarks.py"),
"--out", a.data, "--tasks", ",".join(missing), "--limit", str(max(a.limit,200))])
cmd=[py, os.path.join(TOOLS,"eval_glm.py"), "--snap",a.model,
"--tasks", tasks, "--limit", str(a.limit), "--data", a.data]
if a.ram: cmd+=["--ram",str(a.ram)]
e=env_for(a)
print(f" {C.dim}decode disk-bound: su hardware lento questo richiede ORE. Alza --limit su macchine veloci.{C.r}\n")
print(f" {C.dim}decode is disk-bound: this takes HOURS on slow hardware. Raise --limit on faster machines.{C.r}\n")
sys.exit(subprocess.call(cmd, env=e))
def cmd_convert(a):
@@ -499,34 +499,34 @@ def cmd_convert(a):
"--repo", a.repo, "--outdir", a.model, "--ebits", str(a.ebits), "--io-bits", str(a.io_bits)]
if a.xbits: base+=["--xbits",str(a.xbits)]
# passo 1: modello principale (78 layer). Resumabile: riparte dagli shard mancanti.
print(f" {C.dim}[1/2] modello: {' '.join(base)}{C.r}")
print(f" {C.dim}[1/2] model: {' '.join(base)}{C.r}")
rc=subprocess.call(base)
if rc!=0: sys.exit(rc)
if a.no_mtp: sys.exit(0)
# passo 2: testa MTP (layer 78). SEMPRE int8: a int4 i draft sbagliano quasi sempre
# (acceptance 0-4% vs 39-59%, misurato — issue #8) e la speculazione non parte mai.
mtp_cmd=list(base); i=mtp_cmd.index("--ebits"); mtp_cmd[i+1]=str(max(8,a.ebits))
print(f" {C.dim}[2/2] testa MTP a int8 (draft speculativi){C.r}")
print(f" {C.dim}[2/2] int8 MTP head (speculative drafts){C.r}")
sys.exit(subprocess.call(mtp_cmd+["--mtp"]))
def main():
common=argparse.ArgumentParser(add_help=False)
common.add_argument("--model", default=DEF_MODEL); common.add_argument("--ram", type=int, default=0) # 0 = auto (il motore usa l'88% della RAM disponibile)
common.add_argument("--auto-tier",action="store_true",help="applica automaticamente il piano RAM/VRAM")
common.add_argument("--auto-tier",action="store_true",help="automatically apply the RAM/VRAM plan")
common.add_argument("--ctx",type=int,default=0)
common.add_argument("--gpu",default=None,help="auto, none oppure lista device, es. 0,1")
common.add_argument("--vram",type=float,default=0,help="budget VRAM totale in GB (0=auto)")
common.add_argument("--repin", type=int, default=0, help="adatta gli expert RAM/VRAM ogni N token")
common.add_argument("--gpu",default=None,help="auto, none, or a device list such as 0,1")
common.add_argument("--vram",type=float,default=0,help="total VRAM budget in GB (0=auto)")
common.add_argument("--repin", type=int, default=0, help="adapt RAM/VRAM experts every N tokens")
common.add_argument("--cap", type=int, default=8); common.add_argument("--ngen", type=int, default=1024) # rete di sicurezza: la fine vera la decidono gli stop token
common.add_argument("--topp", type=float, default=0); common.add_argument("--topk", type=int, default=0)
common.add_argument("--temp", type=float, default=None) # temperatura token (0=greedy, default 1.0+nucleus .95)
ap=argparse.ArgumentParser(prog="coli", parents=[common], description="colibrì — GLM-5.2 in locale")
ap=argparse.ArgumentParser(prog="coli", parents=[common], description="colibrì — run GLM-5.2 locally")
sub=ap.add_subparsers(dest="cmd")
sub.add_parser("build", parents=[common]); sub.add_parser("info", parents=[common])
pp=sub.add_parser("plan",parents=[common])
pp.add_argument("--json",action="store_true")
pd=sub.add_parser("doctor",parents=[common])
pd.add_argument("--json",action="store_true",help="emette un report JSON versionato")
pd.add_argument("--json",action="store_true",help="emit a versioned JSON report")
pr=sub.add_parser("run", parents=[common]); pr.add_argument("prompt", nargs="*")
sub.add_parser("chat", parents=[common])
ps=sub.add_parser("serve", parents=[common])
@@ -541,7 +541,7 @@ def main():
pb.add_argument("--limit",type=int,default=40); pb.add_argument("--data",default=os.path.join(HERE,"bench"))
pc=sub.add_parser("convert", parents=[common]); pc.add_argument("--repo",default="zai-org/GLM-5.2-FP8")
pc.add_argument("--ebits",type=int,default=4); pc.add_argument("--io-bits",type=int,default=8); pc.add_argument("--xbits",type=int,default=0)
pc.add_argument("--no-mtp",action="store_true",help="salta la testa MTP (niente draft speculativi)")
pc.add_argument("--no-mtp",action="store_true",help="skip the MTP head (no speculative drafts)")
a=ap.parse_args()
handler={"build":cmd_build,"info":cmd_info,"plan":cmd_plan,"doctor":cmd_doctor,
"run":cmd_run,"chat":cmd_chat,"serve":cmd_serve,"bench":cmd_bench,
c/glm.c
+76 -76
View File
@@ -161,10 +161,10 @@ static int qt_cuda_upload(QT *t){
}
static void cuda_stats_print(void){
size_t n=0,b=0; coli_cuda_stats(-1,&n,&b);
fprintf(stderr,"[CUDA] resident set: %zu tensor, %.2f GB VRAM\n",n,b/1e9);
fprintf(stderr,"[CUDA] resident set: %zu tensors, %.2f GB VRAM\n",n,b/1e9);
if(g_cuda_ndev>1) for(int i=0;i<g_cuda_ndev;i++){
coli_cuda_stats(g_cuda_devices[i],&n,&b);
fprintf(stderr,"[CUDA] device %d: %zu tensor, %.2f GB\n",g_cuda_devices[i],n,b/1e9);
fprintf(stderr,"[CUDA] device %d: %zu tensors, %.2f GB\n",g_cuda_devices[i],n,b/1e9);
}
}
static int parse_cuda_devices(const char *list, int *out){
@@ -195,7 +195,7 @@ static double rss_gb(void){ struct rusage r; getrusage(RUSAGE_SELF,&r);
static float *falloc(int64_t n){
/* guardia anti-wrap (report PR #25): n assurdo da file modello ostili non deve
* diventare una malloc piccola. Niente calloc: il memset nel percorso caldo costa. */
if(n<0 || (uint64_t)n > SIZE_MAX/sizeof(float)){ fprintf(stderr,"falloc: n=%lld fuori range\n",(long long)n); exit(1); }
if(n<0 || (uint64_t)n > SIZE_MAX/sizeof(float)){ fprintf(stderr,"falloc: n=%lld is out of range\n",(long long)n); exit(1); }
float *p=malloc((size_t)n*sizeof(float)); if(!p){fprintf(stderr,"OOM\n");exit(1);} return p; }
/* y[S,O] = x[S,I] @ W^T, W[O,I] f32 */
@@ -481,7 +481,7 @@ static void matmul_qt(float *y, const float *x, QT *w, int S){
: w->fmt==1 ? (const void*)w->q8 : (const void*)w->q4;
if(coli_cuda_matmul(&w->cuda,y,x,weights,w->s,w->fmt,S,w->I,w->O,w->cuda_device)) return;
w->cuda_failed=1;
fprintf(stderr,"[CUDA] tensor [%d,%d] su device %d disabilitato dopo errore; fallback CPU\n",
fprintf(stderr,"[CUDA] tensor [%d,%d] on device %d disabled after an error; falling back to CPU\n",
w->O,w->I,w->cuda_device);
}
#endif
@@ -681,11 +681,11 @@ static void load_cfg(Cfg *c, const char *snap){
} }
c->qk_head=c->qk_nope+c->qk_rope;
c->attn_scale = 1.f / sqrtf((float)c->qk_head);
if(c->n_group!=1){ fprintf(stderr,"questo motore assume n_group=1 (GLM-5.2)\n"); exit(1); }
if(c->n_group!=1){ fprintf(stderr,"this engine requires n_group=1 (GLM-5.2)\n"); exit(1); }
/* VALIDAZIONE (report PR #25): il config.json arriva da mirror non fidati — dimensioni
* ostili non devono superare questo punto. Un solo choke point protegge ogni alloc a valle. */
#define CKR(name,v,lo,hi) if((v)<(lo)||(v)>(hi)){ \
fprintf(stderr,"config: %s=%d fuori range [%d,%d]\n",name,(int)(v),(int)(lo),(int)(hi)); exit(1); }
fprintf(stderr,"config: %s=%d is outside [%d,%d]\n",name,(int)(v),(int)(lo),(int)(hi)); exit(1); }
CKR("hidden_size",c->hidden,1,1<<20) CKR("num_hidden_layers",c->n_layers,1,128)
CKR("num_attention_heads",c->n_heads,1,1024) CKR("n_routed_experts",c->n_experts,1,4096)
CKR("num_experts_per_tok",c->topk,1,64) CKR("moe_intermediate_size",c->moe_inter,1,1<<20)
@@ -729,7 +729,7 @@ static QT qt_load(Model *m, const char *name, int O, int I, int bits){
return t;
}
static float *ld(Model *m, const char *name){ /* tensore 1D f32 residente (norme/bias) */
int64_t n=st_numel(&m->S,name); if(n<0){fprintf(stderr,"manca %s\n",name);exit(1);}
int64_t n=st_numel(&m->S,name); if(n<0){fprintf(stderr,"missing %s\n",name);exit(1);}
float *p=falloc(n); st_read_f32(&m->S,name,p,0); return p;
}
@@ -852,7 +852,7 @@ static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits
m->ix_knw[i]=ld(m,PI("k_norm.weight")); m->ix_knb[i]=ld(m,PI("k_norm.bias"));
#undef PI
}
fprintf(stderr,"[DSA] indexer attivo: attenzione sparsa top-%d oltre %d token di contesto\n",
fprintf(stderr,"[DSA] indexer active: top-%d sparse attention beyond %d context tokens\n",
c->index_topk, c->index_topk);
}
}
@@ -912,7 +912,7 @@ static void expert_load(Model *m, int layer, int eid, ESlot *s){
for(int k=0;k<3;k++){
tw[k]=st_find(&m->S,nm[k]);
snprintf(qn,sizeof(qn),"%s.qs",nm[k]); tq[k]=st_find(&m->S,qn);
if(!tw[k]||!tq[k]){ fprintf(stderr,"manca %s\n",nm[k]); exit(1); }
if(!tw[k]||!tq[k]){ fprintf(stderr,"missing %s\n",nm[k]); exit(1); }
}
int64_t wtot=tw[0]->nbytes+tw[1]->nbytes+tw[2]->nbytes;
int64_t ftot=(tq[0]->nbytes+tq[1]->nbytes+tq[2]->nbytes)/4;
@@ -1548,21 +1548,21 @@ static inline int argmax_v(const float *lo, int V){
static void grammar_setup(Tok *T){
const char *gf=getenv("GRAMMAR"); if(!gf||!*gf) return;
FILE *f=fopen(gf,"rb");
if(!f){ fprintf(stderr,"[GRAMMAR] impossibile aprire %s\n",gf); return; }
if(!f){ fprintf(stderr,"[GRAMMAR] cannot open %s\n",gf); return; }
fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET);
char *txt=malloc((size_t)n+1);
if(!txt || fread(txt,1,(size_t)n,f)!=(size_t)n){
fprintf(stderr,"[GRAMMAR] lettura fallita: %s\n",gf); fclose(f); free(txt); return; }
fprintf(stderr,"[GRAMMAR] failed to read %s\n",gf); fclose(f); free(txt); return; }
fclose(f); txt[n]=0;
if(gr_parse(&g_gram,txt)){ fprintf(stderr,"[GRAMMAR] %s: %s\n",gf,g_gram.err); free(txt); return; }
free(txt);
gr_state_init(&g_gst,&g_gram);
if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammatica non trattabile (ricorsione sinistra?)\n",gf); return; }
if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammar cannot be evaluated (left recursion?)\n",gf); return; }
if(getenv("GRAMMAR_DRAFT")) g_gr_max=atoi(getenv("GRAMMAR_DRAFT"));
if(g_gr_max<1) g_gr_max=1;
if(g_gr_max>48) g_gr_max=48;
g_gr_T=T; g_gr_on=1;
fprintf(stderr,"[GRAMMAR] %s: %d regole, span forzato max %d token/forward\n",gf,g_gram.n,g_gr_max);
fprintf(stderr,"[GRAMMAR] %s: %d rules, forced span capped at %d tokens/forward\n",gf,g_gram.n,g_gr_max);
}
/* stato pulito all'inizio di ogni RISPOSTA (non tra i \x02MORE, che continuano) */
static void grammar_reset(void){
@@ -1592,7 +1592,7 @@ static int grammar_draft(int *draft, int cap){
if(g_gr_prop>=32 && g_gr_acc*2<g_gr_prop){ /* guardia adattiva, come per MTP:
acceptance sotto il 50% = tokenizzazione fuori asse, meglio spegnersi */
g_gr_on=0;
fprintf(stderr,"[GRAMMAR] acceptance %.0f%% dopo %llu proposte: draft grammaticali disattivati\n",
fprintf(stderr,"[GRAMMAR] %.0f%% acceptance after %llu proposals: grammar drafts disabled\n",
100.0*g_gr_acc/g_gr_prop,(unsigned long long)g_gr_prop);
return 0;
}
@@ -1655,7 +1655,7 @@ static void stops_arm(const Cfg *c, int tok_eos){
g_nstop=0;
for(int i=0;i<c->n_stop;i++) g_stop[g_nstop++]=c->stop_ids[i];
if(tok_eos>=0 && !is_stop(tok_eos)) g_stop[g_nstop++]=tok_eos;
fprintf(stderr,"[stop] %d token di stop:",g_nstop);
fprintf(stderr,"[stop] %d stop tokens:",g_nstop);
for(int i=0;i<g_nstop;i++) fprintf(stderr," %d",g_stop[i]);
fprintf(stderr,"\n");
}
@@ -1687,7 +1687,7 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
/* auto-off adattivo: draft che non vengono mai accettati = solo tassa disco */
if(m->has_mtp && m->mtp_prop>=24 && m->mtp_acc*10 < m->mtp_prop){
g_draft=0;
fprintf(stderr,"[MTP] acceptance %.0f%% dopo %llu proposte: draft disattivati\n",
fprintf(stderr,"[MTP] %.0f%% acceptance after %llu proposals: drafts disabled\n",
100.0*m->mtp_acc/m->mtp_prop, (unsigned long long)m->mtp_prop);
}
}
@@ -1703,7 +1703,7 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
float *lo=step_all(m,batch,S,kv); m->n_fw++;
int k=0; /* verifica: accetta finche' coincide */
if(g>0 && getenv("MTP_DEBUG")){ int veri=argmax_v(lo,V);
fprintf(stderr,"[mtpdbg] draft0=%d verita=%d %s\n", draft[0], veri, draft[0]==veri?"HIT":"miss"); }
fprintf(stderr,"[mtpdbg] draft0=%d verified=%d %s\n", draft[0], veri, draft[0]==veri?"HIT":"miss"); }
while(k<g && emitted<n_new){
int accept;
if(g_temp<=0) accept = (argmax_v(lo+(int64_t)k*V,V)==draft[k]);
@@ -1808,8 +1808,8 @@ static void generate(Model *m, const int *prompt, int np, int n_new, int *out){
static void profile_print(Model *m, double elapsed){
double accounted=m->t_edisk+m->t_emm+m->t_attn+m->t_head;
printf("PROFILO: expert-disk %.3fs | expert-matmul %.3fs | attention %.3fs "
"(di cui kvb %.3fs) | lm_head %.3fs | altro %.3fs\n",
printf("PROFILE: expert-disk %.3fs | expert-matmul %.3fs | attention %.3fs "
"(including kvb %.3fs) | lm_head %.3fs | other %.3fs\n",
m->t_edisk,m->t_emm,m->t_attn,m->t_kvb,m->t_head,elapsed-accounted);
}
@@ -1817,7 +1817,7 @@ static void profile_print(Model *m, double elapsed){
* replay the oracle sequence one token at a time. CPU and CUDA therefore see
* identical hidden-state inputs even if their argmax predictions differ. */
static void run_replay(Model *m, const int *full, int nfull, int np){
if(np<2||nfull<=np){ fprintf(stderr,"REPLAY richiede prompt e continuation non vuoti\n"); return; }
if(np<2||nfull<=np){ fprintf(stderr,"REPLAY requires a non-empty prompt and continuation\n"); return; }
kv_alloc(m,nfull+2);
float *logit=step(m,full,np-1,0); free(logit);
m->hits=m->miss=m->ereq=m->gpu_expert_calls=0;
@@ -1827,11 +1827,11 @@ static void run_replay(Model *m, const int *full, int nfull, int np){
logit=step(m,full+i,1,i); free(logit); steps++;
}
double dt=now_s()-t0, tot=m->hits+m->miss;
printf("REPLAY decode: %d token in %.3fs | %.2f tok/s | expert hit %.1f%%\n",
printf("REPLAY decode: %d tokens in %.3fs | %.2f tok/s | expert hit %.1f%%\n",
steps,dt,steps/dt,tot?100.0*m->hits/tot:0.0);
profile_print(m,dt);
#ifdef COLI_CUDA
if(m->gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n",
if(m->gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print();
#endif
@@ -1849,8 +1849,8 @@ static void run_text(Model *m, const char *snap, const char *prompt, int ngen){
* distribuzione int4 e' rumore di quantizzazione */
int cap=(int)strlen(prompt)+16; int *pids=malloc(cap*sizeof(int));
int np=tok_encode(&T,prompt,(int)strlen(prompt),pids,cap);
if(np<1){ fprintf(stderr,"prompt vuoto dopo tokenizzazione\n"); return; }
printf("prompt: %d token | genero fino a %d (stop EOS=%d) | draft n-gram=%d\n", np, ngen, eos, g_draft);
if(np<1){ fprintf(stderr,"prompt is empty after tokenization\n"); return; }
printf("prompt: %d tokens | generating up to %d (EOS stop=%d) | n-gram draft=%d\n", np, ngen, eos, g_draft);
fputs(prompt,stdout); fflush(stdout);
kv_alloc(m, np+ngen+g_draft+2);
int *all=malloc((np+ngen+g_draft+2)*sizeof(int)); memcpy(all,pids,np*sizeof(int));
@@ -1862,24 +1862,24 @@ static void run_text(Model *m, const char *snap, const char *prompt, int ngen){
double dt=now_s()-t;
double tot=m->hits+m->miss;
int nsp=0; for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) nsp++;
printf("\n---\n%d token in %.2fs (%.2f tok/s) | hit-rate expert %.1f%% | RSS %.2f GB\n",
printf("\n---\n%d tokens in %.2fs (%.2f tok/s) | expert hit rate %.1f%% | RSS %.2f GB\n",
produced, dt, produced/dt, tot?100.0*m->hits/tot:0.0, rss_gb());
printf("expert caricati/token: %.1f (per-layer %.2f su %d; baseline topk=%d) | TOPK=%d TOPP=%.2f\n",
printf("experts loaded/token: %.1f (per-layer %.2f across %d; baseline topk=%d) | TOPK=%d TOPP=%.2f\n",
produced?(double)m->ereq/produced:0.0, (produced&&nsp)?(double)m->ereq/produced/nsp:0.0, nsp, c->topk, g_topk, g_topp);
printf("speculazione: %.2f token/forward (%llu fw per %llu tok) | MTP acceptance %.0f%% (%llu/%llu)\n",
printf("speculation: %.2f tokens/forward (%llu forwards per %llu tokens) | MTP acceptance %.0f%% (%llu/%llu)\n",
m->n_fw?(double)m->n_emit/m->n_fw:1.0, (unsigned long long)m->n_fw, (unsigned long long)m->n_emit,
m->mtp_prop?100.0*m->mtp_acc/m->mtp_prop:0.0, (unsigned long long)m->mtp_acc, (unsigned long long)m->mtp_prop);
if(g_gr_prop) printf("grammatica: acceptance %.0f%% (%llu/%llu draft forzati)\n",
if(g_gr_prop) printf("grammar: %.0f%% acceptance (%llu/%llu forced drafts)\n",
100.0*g_gr_acc/g_gr_prop, (unsigned long long)g_gr_acc, (unsigned long long)g_gr_prop);
#ifdef COLI_CUDA
if(m->gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n",
if(m->gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print();
#endif
profile_print(m,dt);
if(g_looka){
const char *nm[3]={"token precedente (=SPEC prefetch)","ingresso layer, salto attention","layer successivo (1 giro di anticipo)"};
printf("LOOKAHEAD routing — recall degli expert veri nel top-8 predetto:\n");
const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"};
printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n");
for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i],
la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]);
}
@@ -1950,11 +1950,11 @@ static void repin_pass(Model *m){
qt_cuda_reset(&s->g); qt_cuda_reset(&s->u); qt_cuda_reset(&s->d);
s->g.cuda_eligible=s->u.cuda_eligible=s->d.cuda_eligible=0;
m->gpu_expert_count--; m->gpu_expert_bytes-=old_gpu;
fprintf(stderr,"[REPIN] upload VRAM fallito; slot degradato a RAM\n");
fprintf(stderr,"[REPIN] VRAM upload failed; slot downgraded to RAM\n");
}
}
#endif
fprintf(stderr,"[REPIN] %s layer %d: esce/out %d (heat=%u) <- entra/in %d (heat=%u) in %.0f ms\n",
fprintf(stderr,"[REPIN] %s layer %d: evict %d (heat=%u) <- admit %d (heat=%u) in %.0f ms\n",
tier,cd[b].l,old,old_heat,cd[b].eid,new_heat,(now_s()-t0)*1e3);
}
for(int l=0;l<m->c.n_layers;l++) if(m->eheat[l]) tier_decay(m->eheat[l],m->c.n_experts);
@@ -2014,11 +2014,11 @@ static int kv_disk_load(Model *m, int *hist, int maxctx){
char mg[8]; int32_t h[8], w[8]; kv_hdr(m,w,0);
if(fread(mg,1,8,f)!=8 || memcmp(mg,KV_MAGIC,8) || fread(h,4,8,f)!=8 ||
h[0]!=w[0]||h[1]!=w[1]||h[2]!=w[2]||h[3]!=w[3]||h[4]!=w[4]||h[5]!=w[5]){
fprintf(stderr,"[KV] .coli_kv di un altro modello/versione: ignorato\n"); fclose(f); return 0; }
fprintf(stderr,"[KV] ignoring .coli_kv from a different model or version\n"); fclose(f); return 0; }
int nrec=h[6];
if(nrec<1){ fclose(f); return 0; }
if(nrec>=maxctx-8-g_draft){
fprintf(stderr,"[KV] conversazione salvata (%d token) piu' grande del contesto: riparto da zero\n",nrec);
fprintf(stderr,"[KV] saved conversation (%d tokens) exceeds the context: starting over\n",nrec);
fclose(f); return 0; }
double t0=now_s();
for(int p=0;p<nrec;p++){
@@ -2034,7 +2034,7 @@ out:
fclose(f);
if(nrec>0){
if(m->has_mtp) m->kv_start[c->n_layers]=-1; /* la finestra MTP riparte da sola */
fprintf(stderr,"[KV] conversazione ripresa dal disco: %d token in %.1fs (niente re-prefill)\n",
fprintf(stderr,"[KV] resumed conversation from disk: %d tokens in %.1fs (no re-prefill)\n",
nrec, now_s()-t0);
}
k->disk_nrec=nrec;
@@ -2074,12 +2074,12 @@ static void run_serve(Model *m, const char *snap){
int templ=getenv("CHAT_TEMPLATE")?atoi(getenv("CHAT_TEMPLATE")):1;
g_kvsave = getenv("KVSAVE")?atoi(getenv("KVSAVE")):1;
int nctx=getenv("KV_SLOTS")?atoi(getenv("KV_SLOTS")):1;
if(nctx<1||nctx>16){ fprintf(stderr,"KV_SLOTS deve essere tra 1 e 16\n"); exit(2); }
if(nctx<1||nctx>16){ fprintf(stderr,"KV_SLOTS must be between 1 and 16\n"); exit(2); }
KVState *initial=m->kv; free(initial->kv_start); free(initial);
ServeCtx *ctx=calloc(nctx,sizeof(ServeCtx));
for(int i=0;i<nctx;i++) serve_ctx_init(m,&ctx[i],snap,i,maxctx);
int active=0; ServeCtx *sc=&ctx[0]; kv_bind(m,&sc->kv);
fprintf(stderr,"[KV] context slots: %d x %d token, projected pool %.2f GB\n",
fprintf(stderr,"[KV] context slots: %d x %d tokens, projected pool %.2f GB\n",
nctx,maxctx,kv_pool_bytes(m,maxctx)/1e9);
#define hist (sc->hist)
#define len (sc->len)
@@ -2232,7 +2232,7 @@ static void stats_dump_q(Model *m, const char *path, int quiet){
for(int i=0;i<=c->n_layers;i++){ if(!m->eusage[i]) continue;
for(int e=0;e<c->n_experts;e++) if(m->eusage[i][e]){ fprintf(f,"%d %d %u\n",i,e,m->eusage[i][e]); tot+=m->eusage[i][e]; nz++; } }
fclose(f); rename(tmp,path);
if(!quiet) fprintf(stderr,"[STATS] %lld selezioni su %lld expert distinti -> %s\n",(long long)tot,(long long)nz,path);
if(!quiet) fprintf(stderr,"[STATS] %lld selections across %lld distinct experts -> %s\n",(long long)tot,(long long)nz,path);
}
static void stats_dump(Model *m, const char *path){ stats_dump_q(m,path,0); }
@@ -2290,11 +2290,11 @@ static void pin_wire(Model *m){
if(mem_wire(s->fslab, fl)==0) wired+=fl; else failed++; }
}
if(failed)
fprintf(stderr,"[PIN] mlock: %.1f GB inchiodati/wired, %ld alloc fallite/failed "
"(alza il limite / raise it: ulimit -l unlimited) in %.0fs\n", wired/1e9, failed, now_s()-t0);
fprintf(stderr,"[PIN] mlock: %.1f GB wired, %ld allocations failed "
"(raise the limit: ulimit -l unlimited) in %.0fs\n", wired/1e9, failed, now_s()-t0);
else
fprintf(stderr,"[PIN] mlock: %.1f GB inchiodati in RAM fisica / wired in physical RAM "
"(niente compressione/no compression) in %.0fs\n", wired/1e9, now_s()-t0);
fprintf(stderr,"[PIN] mlock: %.1f GB wired in physical RAM "
"(no compression) in %.0fs\n", wired/1e9, now_s()-t0);
}
static void pin_load(Model *m, const char *statspath, double gb){
@@ -2329,7 +2329,7 @@ static void pin_load(Model *m, const char *statspath, double gb){
expert_load(m,li,r[a].e,&m->pin[li][slot]);
}
m->resident_bytes += (int64_t)npin*eb;
fprintf(stderr,"[PIN] hot-store: %d expert in RAM (%.1f GB) in %.0fs da %s\n",
fprintf(stderr,"[PIN] hot store: %d experts in RAM (%.1f GB) loaded in %.0fs from %s\n",
npin, npin*eb/1e9, now_s()-t0, statspath);
#ifdef COLI_CUDA
if(g_cuda_enabled && g_cuda_expert_gb>0){
@@ -2378,9 +2378,9 @@ static void pin_load(Model *m, const char *statspath, double gb){
break;
}
}
fprintf(stderr,"[CUDA] hot expert tier: %d/%d expert, VRAM %.2f GB (budget totale %.1f GB)\n",
fprintf(stderr,"[CUDA] hot expert tier: %d/%d experts, VRAM %.2f GB (total budget %.1f GB)\n",
m->gpu_expert_count,npin,m->gpu_expert_bytes/1e9,g_cuda_expert_gb);
for(int i=0;i<g_cuda_ndev;i++) fprintf(stderr,"[CUDA] device %d: %d expert, %.2f GB\n",
for(int i=0;i<g_cuda_ndev;i++) fprintf(stderr,"[CUDA] device %d: %d experts, %.2f GB\n",
g_cuda_devices[i],placed_n[i],placed_b[i]/1e9);
}
#endif
@@ -2445,7 +2445,7 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
int auto_b = ram_gb<=0;
if(auto_b){ ram_gb = g_mem_avail_boot*0.88; /* misurata PRIMA del load: il residente gia'
* allocato viene sottratto sotto, non due volte */
if(ram_gb<4){ fprintf(stderr,"[RAM] MemAvailable illeggibile/troppo bassa, assumo 8 GB\n"); ram_gb=8; } }
if(ram_gb<4){ fprintf(stderr,"[RAM] MemAvailable is unreadable or too low; assuming 8 GB\n"); ram_gb=8; } }
/* slack ONESTO, non forfettario (l'OOM del 2026-07-04 veniva da qui):
* ws[64] slab del working-set (si materializzano TUTTI nel prefill batch-union),
* KV cache a max_ctx, kvb_all della ricostruzione k/v in attention,
@@ -2462,8 +2462,8 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
int capmax = (avail>0 && nsp>0) ? (int)(avail/((double)nsp*eb)) : 0;
if(capmax<1) capmax=1;
if(capmax < m->ecap){
fprintf(stderr,"[RAM_GB=%.1f%s] residente %.1f GB + slack %.1f GB (ws %.1f, KV %dx%d %.1f, kvb %.1f), "
"expert %.1f MB x %d layer -> cap abbassato %d->%d (proiezione picco %.1f GB)\n",
fprintf(stderr,"[RAM_GB=%.1f%s] resident %.1f GB + reserve %.1f GB (ws %.1f, KV %dx%d %.1f, kvb %.1f), "
"experts %.1f MB x %d layers -> cap lowered %d->%d (projected peak %.1f GB)\n",
ram_gb,auto_b?" auto":"",m->resident_bytes/1e9,slack/1e9,ws_b/1e9,
kv_slot_count(),max_ctx,kv_b/1e9,kvb_b/1e9,
eb/1e6, nsp, m->ecap, capmax,
@@ -2482,13 +2482,13 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
m->ecache[i]=realloc(m->ecache[i],(size_t)newcap*sizeof(ESlot));
memset(m->ecache[i]+m->ecap,0,(size_t)(newcap-m->ecap)*sizeof(ESlot));
}
fprintf(stderr,"[RAM_GB=%.1f%s] cap ALZATO %d->%d: il budget lo consente "
"(proiezione picco %.1f GB; CAP_RAISE=0 per disattivare)\n",
fprintf(stderr,"[RAM_GB=%.1f%s] cap raised %d->%d: budget allows it "
"(projected peak %.1f GB; set CAP_RAISE=0 to disable)\n",
ram_gb, auto_b?" auto":"", m->ecap, newcap,
(m->resident_bytes + (double)newcap*nsp*eb + slack)/1e9);
m->ecap=newcap;
} else
fprintf(stderr,"[RAM_GB=%.1f%s] cap=%d ok (proiezione picco %.1f GB)\n", ram_gb, auto_b?" auto":"", m->ecap,
fprintf(stderr,"[RAM_GB=%.1f%s] cap=%d ok (projected peak %.1f GB)\n", ram_gb, auto_b?" auto":"", m->ecap,
(m->resident_bytes + (double)m->ecap*nsp*eb + slack)/1e9);
}
}
@@ -2523,47 +2523,47 @@ int main(int argc, char **argv){
int ebits= argc>2?atoi(argv[2]):8;
int dbits= argc>3?atoi(argv[3]):ebits;
if(getenv("SERVE") && (kv_slot_count()<1 || kv_slot_count()>16)){
fprintf(stderr,"KV_SLOTS deve essere tra 1 e 16\n"); return 2;
fprintf(stderr,"KV_SLOTS must be between 1 and 16\n"); return 2;
}
#ifdef COLI_CUDA
if(getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))){
const char *one=getenv("COLI_GPU"), *many=getenv("COLI_GPUS");
if(one&&many){ fprintf(stderr,"usa COLI_GPU oppure COLI_GPUS, non entrambi\n"); return 2; }
if(one&&many){ fprintf(stderr,"use COLI_GPU or COLI_GPUS, not both\n"); return 2; }
if(many) g_cuda_ndev=parse_cuda_devices(many,g_cuda_devices);
else if(one) g_cuda_ndev=parse_cuda_devices(one,g_cuda_devices);
else { g_cuda_ndev=1; g_cuda_devices[0]=0; }
if(g_cuda_ndev<1){ fprintf(stderr,"COLI_GPUS non valido: usa una lista come 0,1,2\n"); return 2; }
if(g_cuda_ndev<1){ fprintf(stderr,"invalid COLI_GPUS: use a list such as 0,1,2\n"); return 2; }
g_cuda_enabled=coli_cuda_init(g_cuda_devices,g_cuda_ndev);
if(!g_cuda_enabled){ fprintf(stderr,"[CUDA] backend richiesto ma non disponibile\n"); return 2; }
if(!g_cuda_enabled){ fprintf(stderr,"[CUDA] requested backend is unavailable\n"); return 2; }
}
g_cuda_dense=getenv("CUDA_DENSE")?atoi(getenv("CUDA_DENSE")):0;
g_cuda_expert_gb=getenv("CUDA_EXPERT_GB")?atof(getenv("CUDA_EXPERT_GB")):0;
if((getenv("COLI_GPU")||getenv("COLI_GPUS"))&&!g_cuda_enabled){ fprintf(stderr,"COLI_GPU(S) richiede COLI_CUDA=1\n"); return 2; }
if(g_cuda_dense&&!g_cuda_enabled){ fprintf(stderr,"CUDA_DENSE richiede COLI_CUDA=1\n"); return 2; }
if(g_cuda_expert_gb>0 && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB richiede COLI_CUDA=1\n"); return 2; }
if((getenv("COLI_GPU")||getenv("COLI_GPUS"))&&!g_cuda_enabled){ fprintf(stderr,"COLI_GPU(S) requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_dense&&!g_cuda_enabled){ fprintf(stderr,"CUDA_DENSE requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_expert_gb>0 && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_enabled) fprintf(stderr,"[CUDA] mode: routed experts%s\n",g_cuda_dense?" + resident dense tensors":" only (resident dense on CPU)");
#else
if((getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))) ||
getenv("COLI_GPU") || getenv("COLI_GPUS") ||
(getenv("CUDA_DENSE") && atoi(getenv("CUDA_DENSE"))) ||
(getenv("CUDA_EXPERT_GB") && atof(getenv("CUDA_EXPERT_GB"))>0)){
fprintf(stderr,"CUDA richiesto ma questo binario e' CPU-only; ricompila con: make CUDA=1\n");
fprintf(stderr,"CUDA was requested, but this binary is CPU-only; rebuild with: make CUDA=1\n");
return 2;
}
#endif
printf("== Motore C GLM (glm_moe_dsa), cache=%d expert/layer | expert@%d-bit densa@%d-bit | idot: " IDOT_KERNEL " ==\n", cap, ebits, dbits);
printf("== GLM C engine (glm_moe_dsa), cache=%d experts/layer | experts@%d-bit dense@%d-bit | idot: " IDOT_KERNEL " ==\n", cap, ebits, dbits);
g_mem_avail_boot = mem_available_gb();
Model m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits);
if(g_draft<0) g_draft = m.has_mtp ? 3 : 0;
if(getenv("DSA_TOPK")) m.c.index_topk=atoi(getenv("DSA_TOPK")); /* override per test */
printf("caricato in %.2fs | densa residente: %.2f MB | layers=%d experts=%d | MTP %s (draft=%d)\n",
printf("loaded in %.2fs | resident dense: %.2f MB | layers=%d experts=%d | MTP %s (draft=%d)\n",
now_s()-t0, m.resident_bytes/(1024.0*1024.0), m.c.n_layers, m.c.n_experts,
m.has_mtp?"ATTIVA":"assente", g_draft);
m.has_mtp?"ACTIVE":"absent", g_draft);
/* anche su stderr: e' il canale che le UI (coli) mostrano all'utente */
fprintf(stderr,"[MTP] %s (draft=%d)\n", m.has_mtp?"attiva: decodifica speculativa nativa":"assente", g_draft);
fprintf(stderr,"[MTP] %s (draft=%d)\n", m.has_mtp?"active: native speculative decoding":"absent", g_draft);
if(!strncmp(snap,"/mnt/",5))
fprintf(stderr,"ATTENZIONE: il modello e' su %s (filesystem 9p/Windows, lento e fadvise inefficace).\n"
" Per RAM e velocita' tienilo su ext4 (es. /home/...).\n", snap);
fprintf(stderr,"WARNING: the model is on %s (slow 9p/Windows filesystem; fadvise is ineffective).\n"
" Keep it on ext4 (for example, /home/...) for memory efficiency and speed.\n", snap);
/* HOT-STORE: PIN=<statsfile> [PIN_GB=g] -> top expert per frequenza fissi in RAM.
* Va PRIMA di cap_for_ram: i pinnati contano nel residente. */
if(getenv("PIN")) pin_load(&m, getenv("PIN"), getenv("PIN_GB")?atof(getenv("PIN_GB")):10.0);
@@ -2574,7 +2574,7 @@ int main(int argc, char **argv){
int est_ctx = getenv("CTX")?atoi(getenv("CTX")):4096; /* stesso default di run_serve */
snprintf(g_usage_path,sizeof(g_usage_path),"%s/.coli_usage",snap);
int64_t hist = usage_load(&m,g_usage_path);
if(hist>0) fprintf(stderr,"[USAGE] storia expert: %lld selezioni (%s)\n",(long long)hist,g_usage_path);
if(hist>0) fprintf(stderr,"[USAGE] expert history: %lld selections (%s)\n",(long long)hist,g_usage_path);
int autopin = getenv("AUTOPIN")?atoi(getenv("AUTOPIN")):1;
if(!getenv("PIN") && autopin && hist>=5000){
/* quota pin proporzionale alla FIDUCIA nella storia: con pochi dati il pin
@@ -2637,7 +2637,7 @@ int main(int argc, char **argv){
if(pred[i]==tf[i]) ok++;
else fprintf(stderr,"[ORACLE] mismatch pos=%d expected=%d got=%d\n",i,tf[i],pred[i]);
}
printf("PREFILL (teacher-forcing) C vs oracolo: %d/%d posizioni | %.1f pos/s\n",
printf("PREFILL (teacher-forcing) C vs oracle: %d/%d positions | %.1f pos/s\n",
ok,nfull,nfull/tdt);
if(ok<nfull) fprintf(stderr,
"[ORACLE] %d/%d mismatches — run: TF=1 DEBUG_LOGITS=1 for top-5 logit dump\n",
@@ -2651,23 +2651,23 @@ int main(int argc, char **argv){
int *out=malloc((np+n_new)*sizeof(int));
double t=now_s(); generate(&m,prompt,np,n_new,out); double dt=now_s()-t;
int match=0;
printf("\nRiferimento (oracolo): "); for(int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nMotore C GLM : "); for(int i=np;i<nfull;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; }
printf("\nToken coincidenti: %d/%d\n", match, n_new);
printf("\nReference (oracle): "); for(int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nGLM C engine : "); for(int i=np;i<nfull;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; }
printf("\nMatching tokens: %d/%d\n", match, n_new);
double tot=m.hits+m.miss;
printf("Speculazione n-gram (DRAFT=%d): %.2f token/forward (%llu fw per %llu tok)\n",
printf("N-gram speculation (DRAFT=%d): %.2f tokens/forward (%llu forwards per %llu tokens)\n",
g_draft, m.n_fw?(double)m.n_emit/m.n_fw:1.0, (unsigned long long)m.n_fw, (unsigned long long)m.n_emit);
printf("Hit-rate cache expert: %.1f%% (hit=%llu miss=%llu) | RSS: %.2f GB | %.1f tok/s\n",
printf("Expert cache hit rate: %.1f%% (hit=%llu miss=%llu) | RSS: %.2f GB | %.1f tok/s\n",
tot?100.0*m.hits/tot:0.0, (unsigned long long)m.hits, (unsigned long long)m.miss, rss_gb(), n_new/dt);
profile_print(&m,dt);
#ifdef COLI_CUDA
if(m.gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n",
if(m.gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m.gpu_expert_count,m.gpu_expert_bytes/1e9,(unsigned long long)m.gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print();
#endif
if(g_looka){
const char *nm[3]={"token precedente (=SPEC prefetch)","ingresso layer, salto attention","layer successivo (1 giro di anticipo)"};
printf("LOOKAHEAD routing — recall degli expert veri nel top-8 predetto:\n");
const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"};
printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n");
for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i],
la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]);
}
c/grammar.h
+25 -25
View File
@@ -124,12 +124,12 @@ static int gr__lit(Grammar *G, int ri, int ai, const char **pp){
while(*p && *p!='"'){
int b;
if(*p=='\\'){ p++; b=gr__esc(&p);
if(b<0){ snprintf(G->err,sizeof G->err,"escape non valido nel letterale"); return -1; } }
if(b<0){ snprintf(G->err,sizeof G->err,"invalid escape in literal"); return -1; } }
else b=(unsigned char)*p++;
GrSym s; memset(&s,0,sizeof s); s.t=GR_CLS; s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7));
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
}
if(*p!='"'){ snprintf(G->err,sizeof G->err,"letterale non chiuso"); return -1; }
if(*p!='"'){ snprintf(G->err,sizeof G->err,"unterminated literal"); return -1; }
*pp=p+1; return 0;
}
static int gr__cls(Grammar *G, int ri, int ai, const char **pp){
@@ -139,22 +139,22 @@ static int gr__cls(Grammar *G, int ri, int ai, const char **pp){
while(*p && *p!=']'){
int lo, hi;
if(*p=='\\'){ p++; lo=gr__esc(&p);
if(lo<0){ snprintf(G->err,sizeof G->err,"escape non valido nella classe"); return -1; } }
if(lo<0){ snprintf(G->err,sizeof G->err,"invalid escape in character class"); return -1; } }
else lo=(unsigned char)*p++;
hi=lo;
if(*p=='-' && p[1] && p[1]!=']'){
p++;
if(*p=='\\'){ p++; hi=gr__esc(&p);
if(hi<0){ snprintf(G->err,sizeof G->err,"escape non valido nella classe"); return -1; } }
if(hi<0){ snprintf(G->err,sizeof G->err,"invalid escape in character class"); return -1; } }
else hi=(unsigned char)*p++;
}
if(hi<lo){ int t=lo; lo=hi; hi=t; }
for(int b=lo;b<=hi;b++) s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7));
}
if(*p!=']'){ snprintf(G->err,sizeof G->err,"classe non chiusa"); return -1; }
if(*p!=']'){ snprintf(G->err,sizeof G->err,"unterminated character class"); return -1; }
if(neg) for(int i=0;i<32;i++) s.c.bits[i]=(uint8_t)~s.c.bits[i];
*pp=p+1;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
return 0;
}
/* postfisso ? * + sull'ITEM appena letto (simboli [n0, n) dell'alternate corrente).
@@ -179,28 +179,28 @@ static int gr__postfix(Grammar *G, int ri, int ai, int n0, char op){
if(gr__push(G,ri,ai,&R)) goto full; /* l'item nell'alternate diventa R */
return 0;
full:
snprintf(G->err,sizeof G->err,"grammatica troppo grande");
snprintf(G->err,sizeof G->err,"grammar is too large");
return -1;
}
static int gr__alts(Grammar *G, int ri, const char **pp, int depth, int in_group){
if(depth>32){ snprintf(G->err,sizeof G->err,"gruppi troppo annidati"); return -1; }
if(depth>32){ snprintf(G->err,sizeof G->err,"groups are nested too deeply"); return -1; }
const char *p=*pp;
int ai=gr__alt_new(G,ri);
if(ai<0){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(ai<0){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
for(;;){
p=gr__ws(p);
if(!*p){
if(in_group){ snprintf(G->err,sizeof G->err,"manca ')'"); return -1; }
if(in_group){ snprintf(G->err,sizeof G->err,"missing ')'"); return -1; }
break;
}
if(*p==')'){
if(!in_group){ snprintf(G->err,sizeof G->err,"')' inatteso"); return -1; }
if(!in_group){ snprintf(G->err,sizeof G->err,"unexpected ')'"); return -1; }
break;
}
if(*p=='|'){
p++;
ai=gr__alt_new(G,ri);
if(ai<0){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(ai<0){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
continue;
}
int n0=G->r[ri].a[ai].n;
@@ -211,24 +211,24 @@ static int gr__alts(Grammar *G, int ri, const char **pp, int depth, int in_group
} else if(*p=='('){
p++;
int gi=gr__anon(G);
if(gi<0){ snprintf(G->err,sizeof G->err,"grammatica troppo grande"); return -1; }
if(gi<0){ snprintf(G->err,sizeof G->err,"grammar is too large"); return -1; }
if(gr__alts(G,gi,&p,depth+1,1)) return -1;
p=gr__ws(p);
if(*p!=')'){ snprintf(G->err,sizeof G->err,"manca ')'"); return -1; }
if(*p!=')'){ snprintf(G->err,sizeof G->err,"missing ')'"); return -1; }
p++;
GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)gi;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
} else if(gr__idch(*p)){
int nl=gr__idlen(p);
const char *after=gr__ws(p+nl);
if(!in_group && !strncmp(after,"::=",3)) break; /* inizia la prossima regola */
int ref=gr__rule(G,p,nl);
if(ref<0){ snprintf(G->err,sizeof G->err,"troppe regole"); return -1; }
if(ref<0){ snprintf(G->err,sizeof G->err,"too many rules"); return -1; }
p+=nl;
GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)ref;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; }
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
} else {
snprintf(G->err,sizeof G->err,"carattere inatteso '%c'",*p); return -1;
snprintf(G->err,sizeof G->err,"unexpected character '%c'",*p); return -1;
}
p=gr__ws(p);
if(*p=='?'||*p=='*'||*p=='+'){ if(gr__postfix(G,ri,ai,n0,*p)) return -1; p++; }
@@ -244,21 +244,21 @@ static int gr_parse(Grammar *G, const char *src){
p=gr__ws(p);
if(!*p) break;
int nl=gr__idlen(p);
if(nl<=0){ snprintf(G->err,sizeof G->err,"attesa una regola, trovato '%c'",*p); return -1; }
if(nl<=0){ snprintf(G->err,sizeof G->err,"expected a rule, found '%c'",*p); return -1; }
const char *name=p;
const char *q=gr__ws(p+nl);
if(strncmp(q,"::=",3)){ snprintf(G->err,sizeof G->err,"atteso '::=' dopo '%.*s'",nl,name); return -1; }
if(strncmp(q,"::=",3)){ snprintf(G->err,sizeof G->err,"expected '::=' after '%.*s'",nl,name); return -1; }
p=q+3;
int ri=gr__rule(G,name,nl);
if(ri<0){ snprintf(G->err,sizeof G->err,"troppe regole"); return -1; }
if(G->r[ri].n>0){ snprintf(G->err,sizeof G->err,"regola '%.*s' duplicata",nl,name); return -1; }
if(ri<0){ snprintf(G->err,sizeof G->err,"too many rules"); return -1; }
if(G->r[ri].n>0){ snprintf(G->err,sizeof G->err,"duplicate rule '%.*s'",nl,name); return -1; }
if(gr__alts(G,ri,&p,0,0)) return -1;
}
for(int i=0;i<G->n;i++){
if(!strcmp(G->r[i].name,"root")) G->root=i;
if(G->r[i].n==0){ snprintf(G->err,sizeof G->err,"regola '%s' usata ma mai definita",G->r[i].name); return -1; }
if(G->r[i].n==0){ snprintf(G->err,sizeof G->err,"rule '%s' is used but never defined",G->r[i].name); return -1; }
}
if(G->root<0){ snprintf(G->err,sizeof G->err,"manca la regola 'root'"); return -1; }
if(G->root<0){ snprintf(G->err,sizeof G->err,"missing 'root' rule"); return -1; }
return 0;
}
static void gr_free(Grammar *G){
c/iobench.c
+5 -5
View File
@@ -18,26 +18,26 @@
#endif
static double now(){ struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t); return t.tv_sec+t.tv_nsec*1e-9; }
int main(int argc,char**argv){
if(argc<2){fprintf(stderr,"uso: %s file [blkMB] [n] [threads] [direct 0/1]\n",argv[0]);return 1;}
if(argc<2){fprintf(stderr,"usage: %s file [blkMB] [n] [threads] [direct 0/1]\n",argv[0]);return 1;}
long blk=(argc>2?atol(argv[2]):19)*1024*1024;
int n=argc>3?atoi(argv[3]):64;
int nth=argc>4?atoi(argv[4]):8;
int direct=argc>5?atoi(argv[5]):1;
#ifdef O_DIRECT
int fd=open(argv[1],O_RDONLY|(direct?O_DIRECT:0));
if(fd<0 && direct){ fprintf(stderr,"O_DIRECT non disponibile (%s), uso buffered\n",strerror(errno));
if(fd<0 && direct){ fprintf(stderr,"O_DIRECT is unavailable (%s); using buffered I/O\n",strerror(errno));
direct=0; fd=open(argv[1],O_RDONLY); }
#else
int fd=open(argv[1],O_RDONLY); /* macOS: F_NOCACHE ~ O_DIRECT */
#ifdef __APPLE__
if(direct && fd>=0) fcntl(fd,F_NOCACHE,1);
#else
if(direct){ fprintf(stderr,"O_DIRECT non disponibile, uso buffered\n"); direct=0; }
if(direct){ fprintf(stderr,"O_DIRECT is unavailable; using buffered I/O\n"); direct=0; }
#endif
#endif
if(fd<0){perror("open");return 1;}
off_t sz=lseek(fd,0,SEEK_END);
if(sz<blk*2){fprintf(stderr,"file troppo piccolo\n");return 1;}
if(sz<blk*2){fprintf(stderr,"file is too small\n");return 1;}
/* offset random pre-generati (stessi per ogni configurazione: srand fisso).
* 30 bit di rand combinati: su Windows RAND_MAX=32767 e un singolo rand()*4096
* copre solo i primi 134 MB del file (tutti in page cache = misura falsa). */
@@ -55,7 +55,7 @@ int main(int argc,char**argv){
compat_aligned_free(buf); /* su Windows posix_memalign=_aligned_malloc: free() corrompe l'heap */
}
double dt=now()-t0;
printf("%s x%d thread: %d letture x %ldMB = %.1f GB in %.2fs -> %.2f GB/s (%.1f ms/blocco effettivi)\n",
printf("%s x%d threads: %d reads x %ldMB = %.1f GB in %.2fs -> %.2f GB/s (%.1f effective ms/block)\n",
direct?"O_DIRECT":"buffered", nth, n, blk/1024/1024, tot/1e9, dt, tot/1e9/dt, dt/n*1000);
close(fd); free(offs); return 0;
}
c/olmoe.c
+10 -10
View File
@@ -137,7 +137,7 @@ static void load_cfg(Cfg *c, const char *snap) {
static float *load_t(Model *m, const char *name) {
int64_t n = st_numel(&m->S, name);
if (n < 0) { fprintf(stderr, "manca %s\n", name); exit(1); }
if (n < 0) { fprintf(stderr, "missing %s\n", name); exit(1); }
float *p = falloc(n);
st_read_f32(&m->S, name, p, 0); /* densa: niente DONTNEED, resta residente */
return p;
@@ -357,7 +357,7 @@ static int *read_int_array(jval *o, const char *key, int *n_out) {
int main(int argc, char **argv) {
const char *snap = getenv("SNAP");
if (!snap) { fprintf(stderr, "imposta SNAP=<dir snapshot>\n"); return 1; }
if (!snap) { fprintf(stderr, "set SNAP=<snapshot directory>\n"); return 1; }
int cap = argc > 1 ? atoi(argv[1]) : 16;
int bits = argc > 2 ? atoi(argv[2]) : 8;
const char *refpath = argc > 3 ? argv[3] : "ref.json";
@@ -369,9 +369,9 @@ int main(int argc, char **argv) {
int np, nfull; int *prompt = read_int_array(ref,"prompt_ids",&np); int *full = read_int_array(ref,"full_ids",&nfull);
int n_new = nfull - np;
printf("== Motore C streaming, cache = %d expert/layer, expert @ %d-bit ==\n", cap, bits);
printf("== Streaming C engine, cache = %d experts/layer, experts @ %d-bit ==\n", cap, bits);
Model m; model_init(&m, snap, cap, bits);
printf("densa caricata in %.1fs | RSS dopo load densa: %.2f GB\n", m.dense_load_s, rss_gb());
printf("resident weights loaded in %.1fs | RSS after load: %.2f GB\n", m.dense_load_s, rss_gb());
int *out = malloc((np + n_new) * sizeof(int));
double t = now_s();
@@ -379,14 +379,14 @@ int main(int argc, char **argv) {
double dt = now_s() - t;
int match = 0;
printf("\nRiferimento: "); for (int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nMotore C : "); for (int i=np;i<nfull;i++) { printf("%d ", out[i]); if (out[i]==full[i]) match++; }
printf("\nToken coincidenti: %d/%d\n", match, n_new);
printf("\nReference: "); for (int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nC engine : "); for (int i=np;i<nfull;i++) { printf("%d ", out[i]); if (out[i]==full[i]) match++; }
printf("\nMatching tokens: %d/%d\n", match, n_new);
double tot = m.hits + m.miss;
printf("\nRSS PICCO: %.2f GB\n", rss_gb());
printf("Hit-rate cache expert: %.1f%% (hit=%llu miss=%llu)\n", tot?100.0*m.hits/tot:0.0,
printf("\nPEAK RSS: %.2f GB\n", rss_gb());
printf("Expert cache hit rate: %.1f%% (hit=%llu miss=%llu)\n", tot?100.0*m.hits/tot:0.0,
(unsigned long long)m.hits, (unsigned long long)m.miss);
printf("Velocita': %.2f tok/s (%.1fs per %d token)\n", n_new/dt, dt, n_new);
printf("Speed: %.2f tok/s (%.1fs for %d tokens)\n", n_new/dt, dt, n_new);
free(buf); free(arena);
return 0;
}
c/scripts/run.sh
+7 -7
View File
@@ -15,24 +15,24 @@ NGEN="${2:-64}"
cd "$CODE"
# 0) sanity: il modello deve stare su ext4, non su 9p/Windows
case "$DIR" in /mnt/*) echo "ERRORE: $DIR e' su /mnt (9p/Windows). Mettilo su ext4."; exit 1;; esac
case "$DIR" in /mnt/*) echo "ERROR: $DIR is on /mnt (9p/Windows). Move it to ext4."; exit 1;; esac
# 1) se un rsync di spostamento e' ancora vivo, aspettalo
while pgrep -f "rsync.*glm52_i4" >/dev/null 2>&1; do
echo "[1/4] attendo lo spostamento su ext4... ($(du -sh "$DIR" 2>/dev/null | cut -f1))"; sleep 20
echo "[1/4] waiting for the move to ext4... ($(du -sh "$DIR" 2>/dev/null | cut -f1))"; sleep 20
done
echo "[1/4] spostamento completato: $(du -sh "$DIR" | cut -f1), shard $(ls "$DIR"/*.safetensors 2>/dev/null | wc -l)"
echo "[1/4] move complete: $(du -sh "$DIR" | cut -f1), shards $(ls "$DIR"/*.safetensors 2>/dev/null | wc -l)"
# 2) riprende+completa la conversione (ripartibile: salta gli shard gia' fatti)
echo "[2/4] conversione (riprende da dove era): output -> $DIR"
echo "[2/4] conversion (resumes where it stopped): output -> $DIR"
python3 tools/convert_fp8_to_int4.py --repo "$REPO" --outdir "$DIR" --ebits 4 --io-bits 8
# 3) il motore richiede tokenizer.json + config.json nella dir del modello
for f in config.json tokenizer.json; do
[ -f "$DIR/$f" ] || { echo "ERRORE: manca $DIR/$f"; exit 1; }
[ -f "$DIR/$f" ] || { echo "ERROR: missing $DIR/$f"; exit 1; }
done
echo "[3/4] compilo il motore"; make -s glm
echo "[3/4] building the engine"; make -s glm
# 4) generazione reale, con auto-cap dal budget RAM e heartbeat RSS su stderr
echo "[4/4] genero (RAM_GB=$RAM_GB, NGEN=$NGEN)"; echo "------"
echo "[4/4] generating (RAM_GB=$RAM_GB, NGEN=$NGEN)"; echo "------"
SNAP="$DIR" RAM_GB="$RAM_GB" PROMPT="$PROMPT" NGEN="$NGEN" ./glm 64
c/scripts/supervisor.sh
+6 -6
View File
@@ -13,7 +13,7 @@ STALL_S=180 # secondi senza crescita del download -> riavvio
CONVLOG=/tmp/convert_supervised.log
exec 9>"$DIR/.supervisor.lock"
flock -n 9 || { echo "supervisore gia' attivo, esco"; exit 1; }
flock -n 9 || { echo "a supervisor is already running; exiting"; exit 1; }
log(){ echo "[$(date +%H:%M:%S)] $*"; }
@@ -21,16 +21,16 @@ start_conv(){
cd "$CODE"
nohup python3 tools/convert_fp8_to_int4.py --repo zai-org/GLM-5.2-FP8 \
--outdir "$DIR" --ebits 4 --io-bits 8 >> "$CONVLOG" 2>&1 &
log "convertitore avviato (PID $!)"
log "converter started (PID $!)"
}
last_size=-1; stall=0
while :; do
done_n=$(ls "$DIR"/out-*.safetensors 2>/dev/null | wc -l)
if [ "$done_n" -ge "$TOTAL" ]; then log "FATTO: $done_n/$TOTAL shard. Esco."; pkill -f convert_fp8 2>/dev/null; exit 0; fi
if [ "$done_n" -ge "$TOTAL" ]; then log "DONE: $done_n/$TOTAL shards. Exiting."; pkill -f convert_fp8 2>/dev/null; exit 0; fi
if ! pgrep -f convert_fp8 >/dev/null; then
log "convertitore non attivo ($done_n/$TOTAL): lo avvio"
log "converter is not running ($done_n/$TOTAL): starting it"
start_conv; last_size=-1; stall=0; sleep 20; continue
fi
@@ -40,12 +40,12 @@ while :; do
if [ "$size" = "$last_size" ]; then
stall=$((stall+30))
if [ "$stall" -ge "$STALL_S" ]; then
log "download FERMO da ${stall}s a $((size/1000000)) MB ($done_n/$TOTAL): riavvio il convertitore"
log "download stalled for ${stall}s at $((size/1000000)) MB ($done_n/$TOTAL): restarting the converter"
pkill -f convert_fp8; sleep 5
start_conv; last_size=-1; stall=0
fi
else
[ "$last_size" -ge 0 ] && [ "$stall" -ge 60 ] && log "download ripreso ($((size/1000000)) MB)"
[ "$last_size" -ge 0 ] && [ "$stall" -ge 60 ] && log "download resumed ($((size/1000000)) MB)"
last_size=$size; stall=0
fi
else
c/setup.sh
+17 -17
View File
@@ -9,35 +9,35 @@ echo "🐦 colibrì — setup"
UNAME_S=$(uname -s)
# 1) dipendenze
command -v make >/dev/null || { echo "manca make"; exit 1; }
command -v make >/dev/null || { echo "make is missing"; exit 1; }
case "$UNAME_S" in
Darwin)
command -v clang >/dev/null || { echo "manca clang (xcode-select --install)"; exit 1; }
command -v clang >/dev/null || { echo "clang is missing (run: xcode-select --install)"; exit 1; }
echo " clang: $(clang --version | head -1) · $(sysctl -n hw.ncpu) core"
echo -n " OpenMP: "
if [ -f "$(brew --prefix libomp 2>/dev/null)/lib/libomp.dylib" ]; then echo "ok (libomp)"
else echo "libomp assente -> build single-thread (consigliato: brew install libomp)"; fi
else echo "libomp is missing -> single-threaded build (recommended: brew install libomp)"; fi
;;
MINGW*|MSYS*)
command -v gcc >/dev/null || { echo "manca gcc (MinGW-w64). Installa: pacman -S mingw-w64-x86_64-gcc make"; exit 1; }
command -v gcc >/dev/null || { echo "gcc is missing (MinGW-w64). Install: pacman -S mingw-w64-x86_64-gcc make"; exit 1; }
echo " gcc: $(gcc -dumpversion) · MinGW-w64"
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "manca libgomp (pacman -S mingw-w64-x86_64-gcc)"; exit 1; }
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "libgomp is missing (pacman -S mingw-w64-x86_64-gcc)"; exit 1; }
;;
*)
command -v gcc >/dev/null || { echo "manca gcc (es: sudo apt install build-essential)"; exit 1; }
command -v gcc >/dev/null || { echo "gcc is missing (for example: sudo apt install build-essential)"; exit 1; }
echo " gcc: $(gcc -dumpversion) · $(nproc) core"
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "manca (libgomp)"; exit 1; }
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "libgomp is missing"; exit 1; }
;;
esac
# 2) build: nativa (veloce, per QUESTA macchina). Per un binario da distribuire: make portable
echo " compilo (ARCH=${ARCH:-native})…"
echo " building (ARCH=${ARCH:-native})…"
make -s glm ARCH="${ARCH:-native}"
# 3) self-test sull'oracolo tiny, se presente
if [ -d glm_tiny ] && [ -f ref_glm.json ]; then
r=$(SNAP=./glm_tiny TF=1 ./glm 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ posizioni" || true)
echo " self-test motore: ${r:-?} (atteso 32/32)"
r=$(SNAP=./glm_tiny TF=1 ./glm 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ positions" || true)
echo " engine self-test: ${r:-?} (expected 32/32)"
fi
# 4) info macchina (la velocità dipende da QUESTI due numeri, non dalla GPU)
@@ -53,12 +53,12 @@ MINGW*|MSYS*)
ram=$(awk '/MemTotal/{printf "%.0f", $2/1e6}' /proc/meminfo 2>/dev/null || echo "?")
;;
esac
echo " RAM: ${ram} GB (più RAM = più expert in cache = più veloce)"
echo " RAM: ${ram} GB (more RAM = more cached experts = faster inference)"
echo
echo "pronto. Prossimi passi:"
echo " ./coli build # (gia' fatto)"
echo " ./coli convert --model /percorso/NVMe/glm52_i4 # genera il modello int4 (ore)"
echo " ./coli info --model /percorso/NVMe/glm52_i4"
echo " ./coli chat --model /percorso/NVMe/glm52_i4 --ram <GB>"
echo "ready. Next steps:"
echo " ./coli build # already done"
echo " ./coli convert --model /path/on/NVMe/glm52_i4 # generate the int4 model (hours)"
echo " ./coli info --model /path/on/NVMe/glm52_i4"
echo " ./coli chat --model /path/on/NVMe/glm52_i4 --ram <GB>"
echo
echo "IMPORTANTE: tieni il modello su disco VELOCE (NVMe/ext4), MAI su /mnt/c o rete."
echo "IMPORTANT: keep the model on fast storage (NVMe/ext4), never on /mnt/c or a network mount."
c/st.h
+5 -5
View File
@@ -52,7 +52,7 @@ static int st_dtype_code(const char *s) {
if (!strcmp(s, "F32")) return 2;
if (!strcmp(s, "U8")) return 3; /* dati quantizzati (int4 packed / int8) */
if (!strcmp(s, "I8")) return 3;
fprintf(stderr, "dtype non gestito: %s\n", s); exit(1);
fprintf(stderr, "unsupported dtype: %s\n", s); exit(1);
}
static inline float bf16_to_f32(uint16_t h) {
@@ -108,7 +108,7 @@ static void st_init(shards *S, const char *snap_dir) {
while ((e = readdir(d))) {
const char *dot = strrchr(e->d_name, '.');
if (dot && !strcmp(dot, ".safetensors")) { /* model.safetensors o model-0000N-of-... */
if (nf >= ST_MAX_SHARDS) { fprintf(stderr, "troppi shard (>%d): alza ST_MAX_SHARDS\n", ST_MAX_SHARDS); exit(1); }
if (nf >= ST_MAX_SHARDS) { fprintf(stderr, "too many shards (>%d): raise ST_MAX_SHARDS\n", ST_MAX_SHARDS); exit(1); }
snprintf(files[nf++], 1024, "%s/%s", snap_dir, e->d_name);
}
}
@@ -182,7 +182,7 @@ static void st_prefetch(shards *S, const char *name) {
* drop=1 -> consiglia al kernel di scartare le pagine (per gli expert in streaming). */
static int64_t st_read_f32(shards *S, const char *name, float *out, int drop) {
st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); }
if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
void *raw = malloc(t->nbytes);
if (pread(t->fd, raw, t->nbytes, t->off) != t->nbytes) { perror("pread data"); exit(1); }
if (t->dtype == 2) {
@@ -208,7 +208,7 @@ static int64_t st_nbytes(shards *S, const char *name) {
* quantizzati int4/int8 del nostro container (dtype U8). drop=1 -> fadvise DONTNEED. */
static void st_read_raw(shards *S, const char *name, void *out, int drop) {
st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); }
if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
if (pread(t->fd, out, t->nbytes, t->off) != t->nbytes) { perror("pread raw"); exit(1); }
if (drop) posix_fadvise(t->fd, t->off, t->nbytes, POSIX_FADV_DONTNEED);
}
@@ -218,7 +218,7 @@ static void st_read_raw(shards *S, const char *name, void *out, int drop) {
* solo expert richiesto via pread del sotto-range, niente lettura dell'intero blocco. */
static void st_read_slice_f32(shards *S, const char *name, int64_t elem_off, int64_t n_elems, float *out, int drop) {
st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); }
if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
int esz = (t->dtype == 2) ? 4 : 2;
int64_t boff = t->off + elem_off * esz, nb = n_elems * esz;
void *raw = malloc(nb);
c/tests/test_benchmark_cuda_fixture.py
+1 -1
View File
@@ -5,7 +5,7 @@ from tools.benchmark_cuda_fixture import parse_output
SAMPLE = """
REPLAY decode: 4 tokens | 12.34 tok/s
PROFILO: expert-disk 1.25s | expert-matmul 2.50s | attention 0.75s | lm_head 0.10s | altro -0.05s
PROFILE: expert-disk 1.25s | expert-matmul 2.50s | attention 0.75s | lm_head 0.10s | other -0.05s
"""
c/tests/test_cli_output.py
+49
View File
@@ -0,0 +1,49 @@
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent
CLI = HERE / "coli"
class CliOutputLanguageTest(unittest.TestCase):
def run_cli(self, *args):
return subprocess.run(
[sys.executable, str(CLI), *args],
cwd=HERE,
text=True,
capture_output=True,
check=False,
timeout=10,
)
def test_help_is_english(self):
result = self.run_cli("--help")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("run GLM-5.2 locally", result.stdout)
self.assertIn("automatically apply the RAM/VRAM plan", result.stdout)
self.assertNotIn("modello", result.stdout.lower())
self.assertNotIn("motore", result.stdout.lower())
def test_info_status_is_english(self):
with tempfile.TemporaryDirectory() as model:
result = self.run_cli("info", "--model", model)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("config.json is missing", result.stdout)
self.assertIn("disk", result.stdout)
self.assertIn("engine", result.stdout)
def test_missing_model_error_is_english(self):
with tempfile.TemporaryDirectory() as directory:
missing_model = str(Path(directory) / "missing-model")
result = self.run_cli("run", "--model", missing_model, "hello")
self.assertNotEqual(result.returncode, 0)
self.assertIn("model not found", result.stderr)
self.assertIn("set COLI_MODEL or use --model", result.stderr)
if __name__ == "__main__":
unittest.main()
c/tests/test_tok.c
+2 -2
View File
@@ -5,10 +5,10 @@
#include "../tok.h"
int main(int argc, char **argv){
if(argc<2){ fprintf(stderr,"uso: %s tokenizer.json < casi\n",argv[0]); return 1; }
if(argc<2){ fprintf(stderr,"usage: %s tokenizer.json < cases\n",argv[0]); return 1; }
Tok T;
tok_load(&T, argv[1]);
fprintf(stderr,"caricato: vocab_ids=%d specials=%d\n", T.n_ids, T.nsp);
fprintf(stderr,"loaded: vocab_ids=%d specials=%d\n", T.n_ids, T.nsp);
char *line=NULL; size_t cap=0; ssize_t nr;
int pass=0, tot=0, dpass=0;
while((nr=getline(&line,&cap,stdin))>=0){
c/tok.h
+1 -1
View File
@@ -97,7 +97,7 @@ static void tok_load(Tok *T, const char *path){
jval *vocab=json_get(model,"vocab");
jval *merges=json_get(model,"merges");
jval *added=json_get(root,"added_tokens");
if(!vocab||!merges){ fprintf(stderr,"tokenizer.json: manca model.vocab/merges\n"); exit(1); }
if(!vocab||!merges){ fprintf(stderr,"tokenizer.json: missing model.vocab/merges\n"); exit(1); }
/* id massimo per dimensionare id2str */
int maxid=0;
c/tok_unicode.h
+1 -1
View File
@@ -1,4 +1,4 @@
/* GENERATO da tools/gen_unicode.py — non modificare a mano. */
/* GENERATED by tools/gen_unicode.py — do not edit by hand. */
#ifndef TOK_UNICODE_H
#define TOK_UNICODE_H
#include <stdint.h>
c/tools/benchmark_cuda_fixture.py
+2 -2
View File
@@ -11,8 +11,8 @@ from pathlib import Path
SPEED_RE = re.compile(r"REPLAY decode:.*\| ([0-9.]+) tok/s")
PROFILE_RE = re.compile(
r"PROFILO: expert-disk ([0-9.]+)s \| expert-matmul ([0-9.]+)s "
r"\| attention ([0-9.]+)s .* lm_head ([0-9.]+)s \| altro ([0-9.-]+)s"
r"PROFILE: expert-disk ([0-9.]+)s \| expert-matmul ([0-9.]+)s "
r"\| attention ([0-9.]+)s .* lm_head ([0-9.]+)s \| other ([0-9.-]+)s"
)
PROFILE_KEYS = ("disk", "expert_matmul", "attention", "lm_head", "other")
c/tools/convert_fp8_to_int4.py
+32 -31
View File
@@ -139,11 +139,12 @@ def main():
ap.add_argument("--min-free-gb", type=float, default=20.0)
ap.add_argument("--selftest", action="store_true")
ap.add_argument("--mtp", action="store_true",
help="scarica/converte SOLO la testa MTP (model.layers.<n_layers>.*) -> out-mtp-*.safetensors")
help="download and convert ONLY the MTP head (model.layers.<n_layers>.*) -> out-mtp-*.safetensors")
ap.add_argument("--indexer", action="store_true",
help="estrae SOLO i pesi del DSA lightning indexer -> out-idx-*.safetensors. ATTENZIONE: "
"i tensori indexer sono sparsi su ~tutti gli shard: ri-scarica l'intero repo (~756 GB "
"di traffico) per tenerne pochi GB. Resumabile shard per shard. Consigliato --ebits 8.")
help="extract ONLY the DSA lightning-indexer weights -> out-idx-*.safetensors. WARNING: "
"indexer tensors are spread across nearly every shard, so this re-downloads the whole "
"repository (~756 GB of traffic) to retain only a few GB. Resumable per shard. "
"Recommended: --ebits 8.")
a = ap.parse_args()
if a.ebits is None:
# testa MTP a int4 = acceptance ~0-4% (misurato, issue #8): il draft sbaglia sempre
@@ -163,8 +164,8 @@ def main():
q = (w / sc.repeat_interleave(bs,0).repeat_interleave(bs,1)).to(torch.float8_e4m3fn)
deq = (q.to(torch.float32) * sc.repeat_interleave(bs,0).repeat_interleave(bs,1))
rel = (deq - w).abs().mean() / w.abs().mean()
print(f"[selftest fp8 block-dequant] errore relativo medio = {rel:.4f} "
f"({'OK' if rel < 0.05 else 'ALTO'})")
print(f"[selftest fp8 block-dequant] mean relative error = {rel:.4f} "
f"({'OK' if rel < 0.05 else 'HIGH'})")
return
os.makedirs(a.outdir, exist_ok=True)
@@ -178,7 +179,7 @@ def main():
for fn in ["config.json"]:
src = os.path.join(a.indir, fn)
if os.path.exists(src): shutil.copy(src, a.outdir)
print(f"convertito {len(shards)} shard -> {a.outdir}")
print(f"converted {len(shards)} shards -> {a.outdir}")
return
# reale: scarica shard per shard, converte, cancella
@@ -211,7 +212,7 @@ def main():
lock = open(os.path.join(a.outdir, ".convert.lock"), "w")
try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
print("ERRORE: un altro convertitore sta gia' lavorando su questa outdir. Esco."); return
print("ERROR: another converter is already using this output directory. Exiting."); return
# dimensioni note dei file, riempite dopo repo_info: il downloader multi-stream le usa
# per calcolare i confini dei segmenti e per sapere quando un file e' completo.
@@ -280,13 +281,13 @@ def main():
except Exception as ex:
with log_lock:
nres[0] += 1
print(f" [dl] s{t}: {type(ex).__name__} a/at {(s0+done[t])/1e9:.2f} GB: "
f"riprendo/resuming (#{nres[0]})", flush=True)
print(f" [dl] s{t}: {type(ex).__name__} at {(s0+done[t])/1e9:.2f} GB: "
f"resuming (#{nres[0]})", flush=True)
_t.sleep(min(15, 1 + nres[0] // NS))
th = [threading.Thread(target=worker, args=(t,), daemon=True) for t in range(NS)]
for x in th: x.start()
print(f" [dl {_t.strftime('%H:%M:%S')}] connesso/connected: {NS} stream, "
f"{sum(done)/1e9:.2f} di/of {expected/1e9:.2f} GB", flush=True)
print(f" [dl {_t.strftime('%H:%M:%S')}] connected: {NS} streams, "
f"{sum(done)/1e9:.2f} of {expected/1e9:.2f} GB", flush=True)
mark = sum(done); tmark = t0
while any(x.is_alive() for x in th):
_t.sleep(5)
@@ -309,7 +310,7 @@ def main():
os.replace(part, out)
dt = max(_t.time() - t0, 1e-9)
print(f" [dl] {fn}: {expected/1e9:.2f} GB in {dt/60:.1f} min "
f"({expected/dt/1e6:.1f} MB/s medi/avg, {NS} stream, {nres[0]} riprese/resumes)", flush=True)
f"({expected/dt/1e6:.1f} MB/s avg, {NS} streams, {nres[0]} resumes)", flush=True)
return out
def _download_single(url, fn, out, part, expected):
@@ -335,9 +336,9 @@ def main():
cl = r.headers.get("Content-Length")
if cl: expected = have + int(cl)
if have == 0 or nres: # segnale di vita subito / immediate sign of life
print(f" [dl {_t.strftime('%H:%M:%S')}] connesso/connected"
print(f" [dl {_t.strftime('%H:%M:%S')}] connected"
f"{f' @ {have/1e9:.2f} GB' if have else ''}"
f"{f' di/of {expected/1e9:.2f} GB' if expected else ''}", flush=True)
f"{f' of {expected/1e9:.2f} GB' if expected else ''}", flush=True)
with open(part, "ab" if have else "wb") as f:
if not have: f.truncate(0)
while True:
@@ -357,16 +358,16 @@ def main():
except urllib.error.HTTPError as ex:
if ex.code == 416: break # gia' completo / already complete
nres += 1
print(f" [dl] HTTP {ex.code} a/at {have/1e9:.2f} GB: riprendo/resuming (#{nres})", flush=True)
print(f" [dl] HTTP {ex.code} at {have/1e9:.2f} GB: resuming (#{nres})", flush=True)
_t.sleep(min(15, 1 + nres))
except Exception as ex:
nres += 1
print(f" [dl] {type(ex).__name__} a/at {have/1e9:.2f} GB: riprendo/resuming (#{nres})", flush=True)
print(f" [dl] {type(ex).__name__} at {have/1e9:.2f} GB: resuming (#{nres})", flush=True)
_t.sleep(min(15, 1 + nres))
os.replace(part, out)
dt = max(_t.time() - t0, 1e-9); sz = os.path.getsize(out)
print(f" [dl] {fn}: {sz/1e9:.2f} GB in {dt/60:.1f} min "
f"({sz/dt/1e6:.1f} MB/s medi/avg, {nres} riprese/resumes)", flush=True)
f"({sz/dt/1e6:.1f} MB/s avg, {nres} resumes)", flush=True)
return out
from safetensors.numpy import save_file
@@ -380,7 +381,7 @@ def main():
break
except KeyboardInterrupt: raise
except Exception as ex:
w = min(60, 5*(att+1)); print(f"repo_info KO ({type(ex).__name__}): riprovo tra {w}s", flush=True); _t.sleep(w)
w = min(60, 5*(att+1)); print(f"repo_info failed ({type(ex).__name__}); retrying in {w}s", flush=True); _t.sleep(w)
shards = sorted(s.rfilename for s in info.siblings if s.rfilename.endswith(".safetensors"))
for fn in ["config.json", "tokenizer.json", "tokenizer_config.json", "generation_config.json"]:
try: shutil.copy(hf_hub_download(a.repo, fn, local_dir=a.outdir+"/_meta"), a.outdir)
@@ -392,19 +393,19 @@ def main():
f"https://huggingface.co/{a.repo}/resolve/main/model.safetensors.index.json", timeout=30).read())["weight_map"]
pref = f"model.layers.{a.n_layers}."
mtp_shards = sorted(set(v for k, v in idx.items() if k.startswith(pref)))
print(f"[MTP] testa nel layer {a.n_layers}: {len(mtp_shards)} shard da processare: {mtp_shards}")
print(f"[MTP] head at layer {a.n_layers}: {len(mtp_shards)} shards to process: {mtp_shards}")
for i, sh in enumerate(mtp_shards):
outp = os.path.join(a.outdir, f"out-mtp-{i:05d}.safetensors")
if os.path.exists(outp): print(f"[MTP] {outp} gia' fatto"); continue
print(f"[MTP {i+1}/{len(mtp_shards)}] scarico {sh}...", flush=True)
if os.path.exists(outp): print(f"[MTP] {outp} already done"); continue
print(f"[MTP {i+1}/{len(mtp_shards)}] downloading {sh}...", flush=True)
p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_mtp=True)
save_file(out, outp)
os.remove(p)
for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True):
if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB, {len(out)} tensori)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[MTP] FATTO."); return
print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB, {len(out)} tensors)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[MTP] DONE."); return
if a.indexer:
import urllib.request
idx = json.loads(urllib.request.urlopen(
@@ -412,25 +413,25 @@ def main():
idx_shards = sorted(set(v for k, v in idx.items()
if "indexer" in k and 0 <= layer_idx(k) < a.n_layers))
tot_gb = len(idx_shards) * 5.4
print(f"[IDX] pesi indexer su {len(idx_shards)} shard (~{tot_gb:.0f} GB di download totale, resumabile)")
print(f"[IDX] indexer weights across {len(idx_shards)} shards (~{tot_gb:.0f} GB total download, resumable)")
for i, sh in enumerate(idx_shards):
outp = os.path.join(a.outdir, f"out-idx-{i:05d}.safetensors")
if os.path.exists(outp): continue # gia' fatto -> ripartibile
print(f"[IDX {i+1}/{len(idx_shards)}] scarico {sh}...", flush=True)
print(f"[IDX {i+1}/{len(idx_shards)}] downloading {sh}...", flush=True)
p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_idx=True)
if out: save_file(out, outp)
os.remove(p)
for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True):
if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({len(out)} tensori)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[IDX] FATTO."); return
print(f" -> {os.path.basename(outp)} ({len(out)} tensors)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[IDX] DONE."); return
for i, sh in enumerate(shards):
if free_gb(a.outdir) < a.min_free_gb:
print(f"STOP: spazio libero < {a.min_free_gb} GB. Libera spazio e rilancia (riprende)."); break
print(f"STOP: free space is below {a.min_free_gb} GB. Free space and rerun to resume."); break
outp = os.path.join(a.outdir, f"out-{i:05d}.safetensors")
if os.path.exists(outp): continue # gia' fatto -> ripartibile
print(f"[{i+1}/{len(shards)}] scarico {sh} (libero {free_gb(a.outdir):.0f} GB)...", flush=True)
print(f"[{i+1}/{len(shards)}] downloading {sh} ({free_gb(a.outdir):.0f} GB free)...", flush=True)
p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits)
save_file(out, outp)
@@ -439,7 +440,7 @@ def main():
if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB)", flush=True)
shutil.rmtree(tmp, ignore_errors=True)
print("FATTO." if i == len(shards)-1 else "INTERROTTO (rilancia per riprendere).")
print("DONE." if i == len(shards)-1 else "INTERRUPTED (rerun to resume).")
if __name__ == "__main__":
main()
c/tools/download_glm52.py
+6 -6
View File
@@ -28,15 +28,15 @@ def check():
sts = [s for s in info.siblings if s.rfilename.endswith(".safetensors")]
free = shutil.disk_usage(os.path.dirname(DEST) or "/").free
print(f"repo: {REPO}")
print(f" file totali: {len(info.siblings)} ({len(sts)} shard safetensors)")
print(f" dimensione totale: {human(tot)}")
print(f" spazio libero in {DEST}: {human(free)}")
print(f" {'OK: ci sta' if free > tot*1.05 else 'ATTENZIONE: spazio insufficiente'}")
print(f" total files: {len(info.siblings)} ({len(sts)} safetensors shards)")
print(f" total size: {human(tot)}")
print(f" free space in {DEST}: {human(free)}")
print(f" {'OK: enough space' if free > tot*1.05 else 'WARNING: not enough space'}")
def download():
os.makedirs(DEST, exist_ok=True)
free = shutil.disk_usage(DEST).free
print(f"Scarico {REPO} -> {DEST} (libero: {human(free)})")
print(f"Downloading {REPO} -> {DEST} (free: {human(free)})")
# resume_download e' implicito; in caso di interruzione, rilancia e riprende.
snapshot_download(
repo_id=REPO,
@@ -44,7 +44,7 @@ def download():
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.model"],
max_workers=8,
)
print("FATTO. Pesi in:", DEST)
print("DONE. Weights saved in:", DEST)
if __name__ == "__main__":
if "--check" in sys.argv:
c/tools/eval_glm.py
+15 -15
View File
@@ -43,7 +43,7 @@ def load_docs(task, data_dir, limit, seed):
return SMOKE[:limit] if limit else SMOKE
path = os.path.join(data_dir, task + ".jsonl")
if not os.path.exists(path):
sys.exit(f"manca {path} — generalo con: python3 tools/fetch_benchmarks.py --out {data_dir} --tasks {task}")
sys.exit(f"missing {path} — generate it with: python3 tools/fetch_benchmarks.py --out {data_dir} --tasks {task}")
docs = [json.loads(l) for l in open(path) if l.strip()]
random.Random(seed).shuffle(docs)
return docs[:limit] if limit else docs
@@ -84,9 +84,9 @@ def score_accuracy(tasks, meta, perq, lp):
print(f"{t:<18} {n:>4} {100*acc/n:>6.1f}% {100*accn/n:>8.1f}%")
overall.append(100 * accn / n)
for mdl, sc in REFERENCE.get(t, {}).items():
if sc is not None: print(f"{' rif '+mdl:<18} {'':>4} {'':>7} {sc:>8.1f}%")
if sc is not None: print(f"{' ref '+mdl:<18} {'':>4} {'':>7} {sc:>8.1f}%")
if overall:
print(f"\nMEDIA acc_norm: {sum(overall)/len(overall):.1f}% su {len(overall)} task")
print(f"\nMEAN acc_norm: {sum(overall)/len(overall):.1f}% across {len(overall)} tasks")
def main():
ap = argparse.ArgumentParser()
@@ -99,8 +99,8 @@ def main():
ap.add_argument("--cap", type=int, default=64)
ap.add_argument("--bits", default="")
ap.add_argument("--seed", type=int, default=1234)
ap.add_argument("--dry", action="store_true", help="costruisci le richieste e fermati (no motore)")
ap.add_argument("--selftest", action="store_true", help="verifica la matematica dello scoring")
ap.add_argument("--dry", action="store_true", help="build requests and stop without running the engine")
ap.add_argument("--selftest", action="store_true", help="verify the scoring calculations")
a = ap.parse_args()
if a.selftest: # acc/acc_norm con logprob sintetici
@@ -113,32 +113,32 @@ def main():
tk = Tokenizer.from_file(os.path.join(a.snap, "tokenizer.json"))
tasks = [t.strip() for t in a.tasks.split(",") if t.strip()]
docs_by_task = {t: load_docs(t, a.data, a.limit, a.seed) for t in tasks}
for t, d in docs_by_task.items(): print(f"[{t}] {len(d)} domande", file=sys.stderr)
for t, d in docs_by_task.items(): print(f"[{t}] {len(d)} questions", file=sys.stderr)
reqs, meta, perq = build_requests(tk, docs_by_task)
print(f"richieste totali: {len(reqs)} (opzioni)", file=sys.stderr)
print(f"total requests: {len(reqs)} (answer options)", file=sys.stderr)
if a.dry:
for r in reqs[:3]: print(" esempio req:", r[:80], "...", file=sys.stderr)
print("DRY: meccanica ok (tokenizzazione+richieste). Niente motore.", file=sys.stderr); return
for r in reqs[:3]: print(" example request:", r[:80], "...", file=sys.stderr)
print("DRY: request construction and tokenization passed. Engine was not run.", file=sys.stderr); return
req_path = tempfile.mktemp(suffix=".txt")
open(req_path, "w").write("\n".join(reqs) + "\n")
env = dict(os.environ, SNAP=a.snap, SCORE=req_path)
if a.ram: env["RAM_GB"] = str(a.ram)
cmd = [a.glm, str(a.cap)] + a.bits.split()
print("eseguo:", " ".join(cmd), file=sys.stderr)
print("running:", " ".join(cmd), file=sys.stderr)
t0 = time.time()
proc = subprocess.run(cmd, env=env, capture_output=True, text=True)
if proc.returncode != 0:
print("ERRORE motore:\n", proc.stderr[-2000:], file=sys.stderr); sys.exit(1)
print("ENGINE ERROR:\n", proc.stderr[-2000:], file=sys.stderr); sys.exit(1)
lines = [l for l in proc.stdout.strip().splitlines() if l and l[0] in "-0123456789"]
if len(lines) != len(reqs):
print(f"ATTENZIONE: {len(lines)} output vs {len(reqs)} richieste", file=sys.stderr)
print(f"WARNING: {len(lines)} outputs for {len(reqs)} requests", file=sys.stderr)
lp = [float(l.split()[0]) for l in lines]
print(f"(motore: {time.time()-t0:.0f}s){proc.stderr.strip().splitlines()[-1] if proc.stderr.strip() else ''}", file=sys.stderr)
print(f"(engine: {time.time()-t0:.0f}s){proc.stderr.strip().splitlines()[-1] if proc.stderr.strip() else ''}", file=sys.stderr)
score_accuracy(tasks, meta, perq, lp)
print("\nNB: confronta acc_norm col punteggio PUBBLICATO di GLM-5.2 (model card). Se vicino,"
"\n la quantizzazione int4 ha preservato il modello. (riempi REFERENCE in tools/eval_glm.py)")
print("\nNOTE: compare acc_norm with GLM-5.2's PUBLISHED model-card score. A close result"
"\n indicates that int4 quantization preserved quality. (Fill REFERENCE in tools/eval_glm.py.)")
os.remove(req_path)
if __name__ == "__main__":
c/tools/fetch_benchmarks.py
+1 -1
View File
@@ -48,7 +48,7 @@ def main():
from datasets import load_dataset
os.makedirs(a.out, exist_ok=True)
for t in [x.strip() for x in a.tasks.split(",") if x.strip()]:
if t not in TASKS: print("task ignoto:", t); continue
if t not in TASKS: print("unknown task:", t); continue
path, cfg, split, fn = TASKS[t]
ds = load_dataset(path, cfg, split=split)
idx = list(range(len(ds))); random.Random(a.seed).shuffle(idx)
c/tools/gen_unicode.py
+1 -1
View File
@@ -41,7 +41,7 @@ def emit(name, rs):
print("};")
print(f"static const int {name}_n = {len(rs)};\n")
print("/* GENERATO da tools/gen_unicode.py — non modificare a mano. */")
print("/* GENERATED by tools/gen_unicode.py — do not edit by hand. */")
print("#ifndef TOK_UNICODE_H\n#define TOK_UNICODE_H\n#include <stdint.h>\n")
emit("uni_L", L); emit("uni_N", N); emit("uni_S", S)
print("""static int uni_in(const uint32_t t[][2], int n, uint32_t cp){
c/tools/make_glm_oracle.py
+2 -2
View File
@@ -53,7 +53,7 @@ with torch.no_grad():
layer.mlp.gate.e_score_correction_bias.copy_(
torch.linspace(-0.1, 0.1, cfg.n_routed_experts))
print("=== tensori dello state_dict (nomi per il loader C) ===")
print("=== state_dict tensors (names used by the C loader) ===")
for n, p in model.state_dict().items():
print(f" {n:60s} {tuple(p.shape)}")
@@ -76,4 +76,4 @@ print("tf_pred:", tf_pred)
model.save_pretrained("glm_tiny", safe_serialization=True)
json.dump(cfg.to_dict(), open("glm_tiny/config.json", "w"))
json.dump({"prompt_ids": prompt, "full_ids": full, "tf_pred": tf_pred}, open("ref_glm.json", "w"))
print("\nsalvato: glm_tiny/ (pesi+config) e ref_glm.json")
print("\nsaved: glm_tiny/ (weights + config) and ref_glm.json")