Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

* feat: standardize runtime output in English

* test: cover English CLI output
This commit is contained in:
Sidd
2026-07-12 17:08:40 +05:30
committed by GitHub
parent 6e7aa6f92e
commit 2416bc9079
24 changed files with 330 additions and 280 deletions
+4 -4
View File
@@ -1,5 +1,5 @@
<p align="center"> <p align="center">
<img src="assets/colibri.svg" width="500" alt="colibrì — piccolo motore, modello immenso"> <img src="assets/colibri.svg" width="500" alt="colibrì — tiny engine, immense model">
</p> </p>
**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk. **Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
@@ -7,7 +7,7 @@
``` ```
$ ./coli chat $ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU 🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
pronto in 32s · residente 9.9 GB ready in 32s · resident 9.9 GB
ciao! ciao!
◆ Ciao! 😊 Come posso aiutarti oggi? ◆ Ciao! 😊 Come posso aiutarti oggi?
``` ```
@@ -154,7 +154,7 @@ make test-python # run Python tests (requires python)
# Verify (tiny model, 2.4 MB): # Verify (tiny model, 2.4 MB):
pip install torch transformers safetensors huggingface_hub pip install torch transformers safetensors huggingface_hub
python tools/make_glm_oracle.py # generate tiny oracle python tools/make_glm_oracle.py # generate tiny oracle
SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 posizioni" SNAP=./glm_tiny TF=1 ./glm.exe 64 16 16 # expect "32/32 positions"
# Run with real model: # Run with real model:
SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference SNAP=D:\glm52_i4 ./glm.exe 64 4 16 # batch inference
@@ -313,7 +313,7 @@ works against the colibrì OpenAI-compatible server (in review, #21) or any othe
compatible endpoint. Nothing leaves the endpoint you configure. The terminal compatible endpoint. Nothing leaves the endpoint you configure. The terminal
`coli chat` remains the first-class interface. `coli chat` remains the first-class interface.
Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (3040% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache. Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (3040% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental — see below), `CAP_RAISE=0` don't auto-grow the expert cache.
**The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.** **The expert cache auto-sizes to your RAM** (since 2026-07-10): the engine now *raises* the LRU cap to fill your `--ram` budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — **if you benchmarked colibrì before this date, rerun: your numbers were capped.**
+2 -2
View File
@@ -49,7 +49,7 @@
<text x="252" y="62" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace" <text x="252" y="62" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="52" font-weight="bold" fill="#00afaf">colibr&#236;</text> font-size="52" font-weight="bold" fill="#00afaf">colibr&#236;</text>
<text x="252" y="94" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace" <text x="252" y="94" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="19" fill="#808080" font-style="italic">piccolo motore, modello immenso</text> font-size="19" fill="#808080" font-style="italic">tiny engine, immense model</text>
<text x="252" y="122" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace" <text x="252" y="122" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace"
font-size="15" fill="#9a9a9a">GLM-5.2 &#183; 744B MoE &#183; int4 &#183; streaming CPU</text> font-size="15" fill="#9a9a9a">GLM-5.2 &#183; 744B MoE &#183; int4 &#183; streaming CPU</text>
</svg> </svg>

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 3.5 KiB

+1 -1
View File
@@ -14,7 +14,7 @@ ifneq ($(OMPDIR),)
OMPC = -Xclang -fopenmp -I$(OMPDIR)/include OMPC = -Xclang -fopenmp -I$(OMPDIR)/include
OMPL = -L$(OMPDIR)/lib -lomp OMPL = -L$(OMPDIR)/lib -lomp
else else
$(warning libomp non trovato: build SINGLE-THREAD. Per il multithread: brew install libomp) $(warning libomp not found: building single-threaded. For multithreading: brew install libomp)
OMPC = OMPC =
OMPL = OMPL =
endif endif
+59 -59
View File
@@ -1,24 +1,24 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
colibrì — piccolo motore, modello immenso. colibrì — tiny engine, immense model.
CLI per far girare GLM-5.2 (744B) in locale, su CPU, in ~15-26 GB di RAM. Run GLM-5.2 (744B) locally on CPU with roughly 15-26 GB of RAM.
coli chat chat interattiva (carica il modello UNA volta) coli chat interactive chat (loads the model once)
coli serve API HTTP compatibile OpenAI (motore persistente) coli serve OpenAI-compatible HTTP API (persistent engine)
coli run "prompt" generazione singola coli run "prompt" one-shot generation
coli info stato: modello, RAM, disco, config coli info model, RAM, disk, and configuration status
coli plan piano risorse Disk / RAM / VRAM coli plan Disk / RAM / VRAM resource plan
coli doctor diagnosi installazione e piano di esecuzione coli doctor installation and execution-plan diagnostics
coli bench [task...] benchmark di qualità (MMLU/HellaSwag/...) coli bench [task...] quality benchmarks (MMLU/HellaSwag/...)
coli convert converte GLM-5.2-FP8 -> int4 (streaming) coli convert convert GLM-5.2-FP8 to int4, one shard at a time
coli build compila il motore coli build build the engine
Config via env o flag (validi anche dopo il sottocomando): Configuration through environment variables or flags (also valid after the subcommand):
COLI_MODEL=<dir> modello (default /home/vincenzo/glm52_i4) COLI_MODEL=<dir> model directory (default /home/vincenzo/glm52_i4)
--ram N budget RAM in GB (auto-cap cache expert) --ram N RAM budget in GB (automatically sizes the expert cache)
--repin N adatta gli expert RAM/VRAM ogni N token --repin N adapt RAM/VRAM experts every N tokens
--topp P top-p adattivo sugli expert --topk N top-k fisso --topp P adaptive expert top-p --topk N fixed top-k
--ngen N token massimi per risposta --cap N slot cache/layer --ngen N maximum response tokens --cap N cache slots/layer
""" """
import os, sys, subprocess, argparse, json, time, signal, shutil, threading, re, codecs, tempfile, textwrap import os, sys, subprocess, argparse, json, time, signal, shutil, threading, re, codecs, tempfile, textwrap
@@ -82,7 +82,7 @@ def banner(sub=""):
sp=sprite_lines() sp=sprite_lines()
txt=[ txt=[
f"{C.teal}{C.b}colibrì{C.r} {C.dim}v1.0{C.r}", f"{C.teal}{C.b}colibrì{C.r} {C.dim}v1.0{C.r}",
f"{C.dim}piccolo motore, modello immenso{C.r}", f"{C.dim}tiny engine, immense model{C.r}",
f"{C.gray}GLM-5.2 · 744B MoE · int4 · streaming CPU{C.r}", f"{C.gray}GLM-5.2 · 744B MoE · int4 · streaming CPU{C.r}",
f"{C.dgray}{sub}{C.r}" if sub else "", f"{C.dgray}{sub}{C.r}" if sub else "",
"", "",
@@ -100,11 +100,11 @@ def term_w(): return min(shutil.get_terminal_size((80,20)).columns, 100)
def need_model(model): def need_model(model):
if not os.path.isdir(model): if not os.path.isdir(model):
sys.exit(f"{C.yel}modello non trovato:{C.r} {model}\n imposta COLI_MODEL o usa --model") sys.exit(f"{C.yel}model not found:{C.r} {model}\n set COLI_MODEL or use --model")
if not os.path.exists(os.path.join(model,"tokenizer.json")): if not os.path.exists(os.path.join(model,"tokenizer.json")):
sys.exit(f"{C.yel}manca tokenizer.json in {model}{C.r}") sys.exit(f"{C.yel}tokenizer.json is missing from {model}{C.r}")
if not os.path.exists(GLM): if not os.path.exists(GLM):
sys.exit(f"{C.yel}motore non compilato.{C.r} Esegui: coli build") sys.exit(f"{C.yel}engine is not built.{C.r} Run: coli build")
def cuda_binary(): def cuda_binary():
if not os.path.exists(GLM) or sys.platform != "linux": return False if not os.path.exists(GLM) or sys.platform != "linux": return False
@@ -149,7 +149,7 @@ def env_for(a):
ram,ctx,devices,vram=resource_request(a,e) ram,ctx,devices,vram=resource_request(a,e)
plan=build_plan(a.model,ram,ctx,devices,vram) plan=build_plan(a.model,ram,ctx,devices,vram)
except (OSError,ValueError,json.JSONDecodeError) as error: except (OSError,ValueError,json.JSONDecodeError) as error:
sys.exit(f"{C.yel}piano risorse non valido:{C.r} {error}") sys.exit(f"{C.yel}invalid resource plan:{C.r} {error}")
has_cuda=cuda_binary() has_cuda=cuda_binary()
e=environment_for_plan(plan,e,has_cuda) e=environment_for_plan(plan,e,has_cuda)
rt=plan["tiers"]["ram"]; vt=plan["tiers"]["vram"] rt=plan["tiers"]["ram"]; vt=plan["tiers"]["vram"]
@@ -292,26 +292,26 @@ def cmd_info(a):
def row(k,v): print(f" {C.gray}{k:<10}{C.r} {v}") def row(k,v): print(f" {C.gray}{k:<10}{C.r} {v}")
if os.path.exists(cfgp): if os.path.exists(cfgp):
c=json.load(open(cfgp)) c=json.load(open(cfgp))
row("modello", a.model) row("model", a.model)
row("arch", f"hidden {c.get('hidden_size')} · {c.get('num_hidden_layers')} layer · " row("arch", f"hidden {c.get('hidden_size')} · {c.get('num_hidden_layers')} layer · "
f"{c.get('n_routed_experts')} expert/layer · top-{c.get('num_experts_per_tok')}") f"{c.get('n_routed_experts')} expert/layer · top-{c.get('num_experts_per_tok')}")
sts=[x for x in os.listdir(a.model) if x.endswith('.safetensors')] sts=[x for x in os.listdir(a.model) if x.endswith('.safetensors')]
sz=sum(os.path.getsize(os.path.join(a.model,x)) for x in sts) sz=sum(os.path.getsize(os.path.join(a.model,x)) for x in sts)
row("shard", f"{len(sts)} file · {sz/1e9:.0f} GB su disco") row("shards", f"{len(sts)} files · {sz/1e9:.0f} GB on disk")
else: else:
print(f" {C.yel}config.json non presente (conversione incompleta?){C.r}") print(f" {C.yel}config.json is missing (incomplete conversion?){C.r}")
try: try:
mi=open('/proc/meminfo').read() mi=open('/proc/meminfo').read()
tot=int(re.search(r'MemTotal:\s+(\d+)',mi).group(1))/1e6 tot=int(re.search(r'MemTotal:\s+(\d+)',mi).group(1))/1e6
av=int(re.search(r'MemAvailable:\s+(\d+)',mi).group(1))/1e6 av=int(re.search(r'MemAvailable:\s+(\d+)',mi).group(1))/1e6
row("RAM", f"{tot:.0f} GB totali · {av:.1f} GB disponibili") row("RAM", f"{tot:.0f} GB total · {av:.1f} GB available")
except Exception: pass except Exception: pass
try: try:
fs = shutil.disk_usage(a.model if os.path.isdir(a.model) else HERE) fs = shutil.disk_usage(a.model if os.path.isdir(a.model) else HERE)
row("disco", f"{fs.free/1e9:.0f} GB liberi") row("disk", f"{fs.free/1e9:.0f} GB free")
except OSError: except OSError:
row("disco", "? GB (non disponibile)") row("disk", "? GB (unavailable)")
row("motore", "pronto ✓" if os.path.exists(GLM) else "da compilare (coli build)") row("engine", "ready ✓" if os.path.exists(GLM) else "not built (coli build)")
knobs=[] knobs=[]
if a.ram: knobs.append(f"ram {a.ram}GB") if a.ram: knobs.append(f"ram {a.ram}GB")
if a.topp: knobs.append(f"topp {a.topp}") if a.topp: knobs.append(f"topp {a.topp}")
@@ -323,11 +323,11 @@ def cmd_plan(a):
from resource_plan import build_plan, format_plan from resource_plan import build_plan, format_plan
try: try:
ram,ctx,devices,vram=resource_request(a,os.environ) ram,ctx,devices,vram=resource_request(a,os.environ)
if ctx<1: raise ValueError("--ctx deve essere positivo") if ctx<1: raise ValueError("--ctx must be positive")
if a.vram<0: raise ValueError("--vram non puo essere negativo") if a.vram<0: raise ValueError("--vram cannot be negative")
plan=build_plan(a.model,ram,ctx,devices,vram) plan=build_plan(a.model,ram,ctx,devices,vram)
except (OSError, ValueError, json.JSONDecodeError) as error: except (OSError, ValueError, json.JSONDecodeError) as error:
sys.exit(f"{C.yel}impossibile creare il piano:{C.r} {error}") sys.exit(f"{C.yel}cannot create resource plan:{C.r} {error}")
if a.json: if a.json:
print(json.dumps(plan,indent=2)) print(json.dumps(plan,indent=2))
return return
@@ -339,9 +339,9 @@ def cmd_doctor(a):
from doctor import exit_code, format_doctor, run_doctor from doctor import exit_code, format_doctor, run_doctor
try: try:
ram,ctx,devices,vram=resource_request(a,os.environ) ram,ctx,devices,vram=resource_request(a,os.environ)
if ctx<1: raise ValueError("--ctx deve essere positivo") if ctx<1: raise ValueError("--ctx must be positive")
if ram<0: raise ValueError("--ram non puo essere negativo") if ram<0: raise ValueError("--ram cannot be negative")
if vram<0: raise ValueError("--vram non puo essere negativo") if vram<0: raise ValueError("--vram cannot be negative")
except ValueError as error: except ValueError as error:
report={"schema_version":1,"status":"error","model":os.path.abspath(a.model), report={"schema_version":1,"status":"error","model":os.path.abspath(a.model),
"checks":[{"id":"config.arguments","status":"fail","summary":str(error)}], "checks":[{"id":"config.arguments","status":"fail","summary":str(error)}],
@@ -354,7 +354,7 @@ def cmd_doctor(a):
def cmd_run(a): def cmd_run(a):
need_model(a.model) need_model(a.model)
prompt=" ".join(a.prompt) if a.prompt else sys.exit('uso: coli run "il tuo prompt"') prompt=" ".join(a.prompt) if a.prompt else sys.exit('usage: coli run "your prompt"')
banner("run") banner("run")
# template ufficiale GLM-5.2: niente \n dopo i ruoli; <think></think> = risposta diretta (nothink) # template ufficiale GLM-5.2: niente \n dopo i ruoli; <think></think> = risposta diretta (nothink)
e=env_for(a); e["PROMPT"]=f"[gMASK]<sop><|user|>{prompt}<|assistant|><think></think>" e=env_for(a); e["PROMPT"]=f"[gMASK]<sop><|user|>{prompt}<|assistant|><think></think>"
@@ -367,24 +367,24 @@ def cmd_chat(a):
e=env_for(a); e["SERVE"]="1" e=env_for(a); e["SERVE"]="1"
p=subprocess.Popen([GLM,str(a.cap)], env=e, stdin=subprocess.PIPE, p=subprocess.Popen([GLM,str(a.cap)], env=e, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=errlog, bufsize=0) stdout=subprocess.PIPE, stderr=errlog, bufsize=0)
sp=Spinner("sveglio il gigante (744B)…"); sp.start() sp=Spinner("waking the giant (744B)…"); sp.start()
st=stream_turn(p, READY, lambda b: None) st=stream_turn(p, READY, lambda b: None)
sp.stop() sp.stop()
if st is None: if st is None:
errlog.seek(0); print(errlog.read()[-1500:]); sys.exit("il motore è uscito durante il load") errlog.seek(0); print(errlog.read()[-1500:]); sys.exit("the engine exited while loading")
errlog.flush() errlog.flush()
try: try:
elog=open(errlog.name).read() elog=open(errlog.name).read()
mload=re.search(r"caricato in ([0-9.]+)s \| densa residente: ([0-9.]+) MB", elog) mload=re.search(r"loaded in ([0-9.]+)s \| resident dense: ([0-9.]+) MB", elog)
if mload: print(f" {C.grn}✓{C.r} pronto in {mload.group(1)}s {C.dim}· residente {float(mload.group(2))/1000:.1f} GB · RSS {st.get('rss','?')} GB{C.r}") if mload: print(f" {C.grn}✓{C.r} ready in {mload.group(1)}s {C.dim}· resident {float(mload.group(2))/1000:.1f} GB · RSS {st.get('rss','?')} GB{C.r}")
for l in elog.splitlines(): # una riga di stato per riga, senza path for l in elog.splitlines(): # una riga di stato per riga, senza path
if l.startswith(("[RAM_GB","[PIN]","[MTP]","[USAGE]","[DSA]","[KV]")): if l.startswith(("[RAM_GB","[PIN]","[MTP]","[USAGE]","[DSA]","[KV]")):
l=re.sub(r" ?\(?/[^ )]+\)?","",l.strip()) # via i percorsi lunghi l=re.sub(r" ?\(?/[^ )]+\)?","",l.strip()) # via i percorsi lunghi
l=re.sub(r" da$| in$","",l) l=re.sub(r" from$","",l)
for chunk in textwrap.wrap(l, term_w()-4) or [l]: for chunk in textwrap.wrap(l, term_w()-4) or [l]:
print(f" {C.dgray}{chunk}{C.r}") print(f" {C.dgray}{chunk}{C.r}")
except Exception: pass except Exception: pass
print(f" {C.dim}scrivi e premi invio · :piu continua risposta · :reset memoria · :q esci{C.r}\n") print(f" {C.dim}type and press Enter · :more continues · :reset clears memory · :q exits{C.r}\n")
w=term_w()-4 w=term_w()-4
def user_box(msg): def user_box(msg):
"""ri-disegna il messaggio dentro una box che si ADATTA su piu' righe: """ri-disegna il messaggio dentro una box che si ADATTA su piu' righe:
@@ -415,7 +415,7 @@ def cmd_chat(a):
if msg==":reset": if msg==":reset":
p.stdin.write(b"\x02RESET\n"); p.stdin.flush() p.stdin.write(b"\x02RESET\n"); p.stdin.flush()
stream_turn(p, END, lambda b: None) stream_turn(p, END, lambda b: None)
print(f" {C.dim}✦ memoria azzerata{C.r}\n"); continue print(f" {C.dim}✦ memory cleared{C.r}\n"); continue
if msg in (":piu",":più",":more",":continua"): if msg in (":piu",":più",":more",":continua"):
p.stdin.write(b"\x02MORE\n"); p.stdin.flush() p.stdin.write(b"\x02MORE\n"); p.stdin.flush()
else: else:
@@ -430,7 +430,7 @@ def cmd_chat(a):
pl=[l for l in tail.splitlines() if l.startswith("[prefill]")] pl=[l for l in tail.splitlines() if l.startswith("[prefill]")]
return pl[-1].replace("[prefill] ","prefill ") if pl else "" return pl[-1].replace("[prefill] ","prefill ") if pl else ""
except Exception: return "" except Exception: return ""
sp2=Spinner("pensa…", tick=prefill_tick); sp2.start() sp2=Spinner("thinking…", tick=prefill_tick); sp2.start()
md=MDStream(" ") # markdown -> terminale, in streaming md=MDStream(" ") # markdown -> terminale, in streaming
raw=os.environ.get("COLI_RAW")=="1" raw=os.environ.get("COLI_RAW")=="1"
def echo(bs, _dec=dec, _st=state): def echo(bs, _dec=dec, _st=state):
@@ -445,23 +445,23 @@ def cmd_chat(a):
st=stream_turn(p, END, echo) st=stream_turn(p, END, echo)
if not raw: md.close() if not raw: md.close()
sp2.stop() sp2.stop()
if st is None: print(f"\n {C.yel}[motore terminato]{C.r}"); break if st is None: print(f"\n {C.yel}[engine terminated]{C.r}"); break
el=time.time()-t0 el=time.time()-t0
if st.get("tok"): if st.get("tok"):
print(f"\r {C.dgray}└─ {st['tok']} tok · {st['tps']:.2f} tok/s · hit {st['hit']:.0f}% · RSS {st['rss']:.1f} GB · {el:.0f}s{C.r}") print(f"\r {C.dgray}└─ {st['tok']} tok · {st['tps']:.2f} tok/s · hit {st['hit']:.0f}% · RSS {st['rss']:.1f} GB · {el:.0f}s{C.r}")
if st["tok"]>=a.ngen: if st["tok"]>=a.ngen:
print(f" {C.yel}…troncato al limite --ngen ({a.ngen}): scrivi :piu per far continuare la risposta{C.r}") print(f" {C.yel}…stopped at --ngen ({a.ngen}); type :more to continue the response{C.r}")
print() print()
else: else:
print() print()
except KeyboardInterrupt: except KeyboardInterrupt:
print(f"\n {C.dim}interrotto{C.r}") print(f"\n {C.dim}interrupted{C.r}")
finally: finally:
try: p.stdin.close(); p.terminate() try: p.stdin.close(); p.terminate()
except Exception: pass except Exception: pass
try: os.unlink(errlog.name) try: os.unlink(errlog.name)
except Exception: pass except Exception: pass
print(f" {C.teal}ciao{C.r} {C.dim}— il colibrì torna al nido{C.r} 🐦\n") print(f" {C.teal}goodbye{C.r} {C.dim}— the hummingbird returns to its nest{C.r} 🐦\n")
def cmd_serve(a): def cmd_serve(a):
need_model(a.model) need_model(a.model)
@@ -480,14 +480,14 @@ def cmd_bench(a):
# dataset mancanti -> li scarica una volta (fetch_benchmarks.py li mette in --data come JSONL) # dataset mancanti -> li scarica una volta (fetch_benchmarks.py li mette in --data come JSONL)
missing=[t for t in tasks.split(",") if not os.path.exists(os.path.join(a.data,f"{t}.jsonl"))] missing=[t for t in tasks.split(",") if not os.path.exists(os.path.join(a.data,f"{t}.jsonl"))]
if missing: if missing:
print(f" {C.dim}scarico i dataset mancanti: {', '.join(missing)}{C.r}") print(f" {C.dim}downloading missing datasets: {', '.join(missing)}{C.r}")
subprocess.call([py, os.path.join(TOOLS,"fetch_benchmarks.py"), subprocess.call([py, os.path.join(TOOLS,"fetch_benchmarks.py"),
"--out", a.data, "--tasks", ",".join(missing), "--limit", str(max(a.limit,200))]) "--out", a.data, "--tasks", ",".join(missing), "--limit", str(max(a.limit,200))])
cmd=[py, os.path.join(TOOLS,"eval_glm.py"), "--snap",a.model, cmd=[py, os.path.join(TOOLS,"eval_glm.py"), "--snap",a.model,
"--tasks", tasks, "--limit", str(a.limit), "--data", a.data] "--tasks", tasks, "--limit", str(a.limit), "--data", a.data]
if a.ram: cmd+=["--ram",str(a.ram)] if a.ram: cmd+=["--ram",str(a.ram)]
e=env_for(a) e=env_for(a)
print(f" {C.dim}decode disk-bound: su hardware lento questo richiede ORE. Alza --limit su macchine veloci.{C.r}\n") print(f" {C.dim}decode is disk-bound: this takes HOURS on slow hardware. Raise --limit on faster machines.{C.r}\n")
sys.exit(subprocess.call(cmd, env=e)) sys.exit(subprocess.call(cmd, env=e))
def cmd_convert(a): def cmd_convert(a):
@@ -499,34 +499,34 @@ def cmd_convert(a):
"--repo", a.repo, "--outdir", a.model, "--ebits", str(a.ebits), "--io-bits", str(a.io_bits)] "--repo", a.repo, "--outdir", a.model, "--ebits", str(a.ebits), "--io-bits", str(a.io_bits)]
if a.xbits: base+=["--xbits",str(a.xbits)] if a.xbits: base+=["--xbits",str(a.xbits)]
# passo 1: modello principale (78 layer). Resumabile: riparte dagli shard mancanti. # passo 1: modello principale (78 layer). Resumabile: riparte dagli shard mancanti.
print(f" {C.dim}[1/2] modello: {' '.join(base)}{C.r}") print(f" {C.dim}[1/2] model: {' '.join(base)}{C.r}")
rc=subprocess.call(base) rc=subprocess.call(base)
if rc!=0: sys.exit(rc) if rc!=0: sys.exit(rc)
if a.no_mtp: sys.exit(0) if a.no_mtp: sys.exit(0)
# passo 2: testa MTP (layer 78). SEMPRE int8: a int4 i draft sbagliano quasi sempre # passo 2: testa MTP (layer 78). SEMPRE int8: a int4 i draft sbagliano quasi sempre
# (acceptance 0-4% vs 39-59%, misurato — issue #8) e la speculazione non parte mai. # (acceptance 0-4% vs 39-59%, misurato — issue #8) e la speculazione non parte mai.
mtp_cmd=list(base); i=mtp_cmd.index("--ebits"); mtp_cmd[i+1]=str(max(8,a.ebits)) mtp_cmd=list(base); i=mtp_cmd.index("--ebits"); mtp_cmd[i+1]=str(max(8,a.ebits))
print(f" {C.dim}[2/2] testa MTP a int8 (draft speculativi){C.r}") print(f" {C.dim}[2/2] int8 MTP head (speculative drafts){C.r}")
sys.exit(subprocess.call(mtp_cmd+["--mtp"])) sys.exit(subprocess.call(mtp_cmd+["--mtp"]))
def main(): def main():
common=argparse.ArgumentParser(add_help=False) common=argparse.ArgumentParser(add_help=False)
common.add_argument("--model", default=DEF_MODEL); common.add_argument("--ram", type=int, default=0) # 0 = auto (il motore usa l'88% della RAM disponibile) common.add_argument("--model", default=DEF_MODEL); common.add_argument("--ram", type=int, default=0) # 0 = auto (il motore usa l'88% della RAM disponibile)
common.add_argument("--auto-tier",action="store_true",help="applica automaticamente il piano RAM/VRAM") common.add_argument("--auto-tier",action="store_true",help="automatically apply the RAM/VRAM plan")
common.add_argument("--ctx",type=int,default=0) common.add_argument("--ctx",type=int,default=0)
common.add_argument("--gpu",default=None,help="auto, none oppure lista device, es. 0,1") common.add_argument("--gpu",default=None,help="auto, none, or a device list such as 0,1")
common.add_argument("--vram",type=float,default=0,help="budget VRAM totale in GB (0=auto)") common.add_argument("--vram",type=float,default=0,help="total VRAM budget in GB (0=auto)")
common.add_argument("--repin", type=int, default=0, help="adatta gli expert RAM/VRAM ogni N token") common.add_argument("--repin", type=int, default=0, help="adapt RAM/VRAM experts every N tokens")
common.add_argument("--cap", type=int, default=8); common.add_argument("--ngen", type=int, default=1024) # rete di sicurezza: la fine vera la decidono gli stop token common.add_argument("--cap", type=int, default=8); common.add_argument("--ngen", type=int, default=1024) # rete di sicurezza: la fine vera la decidono gli stop token
common.add_argument("--topp", type=float, default=0); common.add_argument("--topk", type=int, default=0) common.add_argument("--topp", type=float, default=0); common.add_argument("--topk", type=int, default=0)
common.add_argument("--temp", type=float, default=None) # temperatura token (0=greedy, default 1.0+nucleus .95) common.add_argument("--temp", type=float, default=None) # temperatura token (0=greedy, default 1.0+nucleus .95)
ap=argparse.ArgumentParser(prog="coli", parents=[common], description="colibrì — GLM-5.2 in locale") ap=argparse.ArgumentParser(prog="coli", parents=[common], description="colibrì — run GLM-5.2 locally")
sub=ap.add_subparsers(dest="cmd") sub=ap.add_subparsers(dest="cmd")
sub.add_parser("build", parents=[common]); sub.add_parser("info", parents=[common]) sub.add_parser("build", parents=[common]); sub.add_parser("info", parents=[common])
pp=sub.add_parser("plan",parents=[common]) pp=sub.add_parser("plan",parents=[common])
pp.add_argument("--json",action="store_true") pp.add_argument("--json",action="store_true")
pd=sub.add_parser("doctor",parents=[common]) pd=sub.add_parser("doctor",parents=[common])
pd.add_argument("--json",action="store_true",help="emette un report JSON versionato") pd.add_argument("--json",action="store_true",help="emit a versioned JSON report")
pr=sub.add_parser("run", parents=[common]); pr.add_argument("prompt", nargs="*") pr=sub.add_parser("run", parents=[common]); pr.add_argument("prompt", nargs="*")
sub.add_parser("chat", parents=[common]) sub.add_parser("chat", parents=[common])
ps=sub.add_parser("serve", parents=[common]) ps=sub.add_parser("serve", parents=[common])
@@ -541,7 +541,7 @@ def main():
pb.add_argument("--limit",type=int,default=40); pb.add_argument("--data",default=os.path.join(HERE,"bench")) pb.add_argument("--limit",type=int,default=40); pb.add_argument("--data",default=os.path.join(HERE,"bench"))
pc=sub.add_parser("convert", parents=[common]); pc.add_argument("--repo",default="zai-org/GLM-5.2-FP8") pc=sub.add_parser("convert", parents=[common]); pc.add_argument("--repo",default="zai-org/GLM-5.2-FP8")
pc.add_argument("--ebits",type=int,default=4); pc.add_argument("--io-bits",type=int,default=8); pc.add_argument("--xbits",type=int,default=0) pc.add_argument("--ebits",type=int,default=4); pc.add_argument("--io-bits",type=int,default=8); pc.add_argument("--xbits",type=int,default=0)
pc.add_argument("--no-mtp",action="store_true",help="salta la testa MTP (niente draft speculativi)") pc.add_argument("--no-mtp",action="store_true",help="skip the MTP head (no speculative drafts)")
a=ap.parse_args() a=ap.parse_args()
handler={"build":cmd_build,"info":cmd_info,"plan":cmd_plan,"doctor":cmd_doctor, handler={"build":cmd_build,"info":cmd_info,"plan":cmd_plan,"doctor":cmd_doctor,
"run":cmd_run,"chat":cmd_chat,"serve":cmd_serve,"bench":cmd_bench, "run":cmd_run,"chat":cmd_chat,"serve":cmd_serve,"bench":cmd_bench,
+76 -76
View File
@@ -161,10 +161,10 @@ static int qt_cuda_upload(QT *t){
} }
static void cuda_stats_print(void){ static void cuda_stats_print(void){
size_t n=0,b=0; coli_cuda_stats(-1,&n,&b); size_t n=0,b=0; coli_cuda_stats(-1,&n,&b);
fprintf(stderr,"[CUDA] resident set: %zu tensor, %.2f GB VRAM\n",n,b/1e9); fprintf(stderr,"[CUDA] resident set: %zu tensors, %.2f GB VRAM\n",n,b/1e9);
if(g_cuda_ndev>1) for(int i=0;i<g_cuda_ndev;i++){ if(g_cuda_ndev>1) for(int i=0;i<g_cuda_ndev;i++){
coli_cuda_stats(g_cuda_devices[i],&n,&b); coli_cuda_stats(g_cuda_devices[i],&n,&b);
fprintf(stderr,"[CUDA] device %d: %zu tensor, %.2f GB\n",g_cuda_devices[i],n,b/1e9); fprintf(stderr,"[CUDA] device %d: %zu tensors, %.2f GB\n",g_cuda_devices[i],n,b/1e9);
} }
} }
static int parse_cuda_devices(const char *list, int *out){ static int parse_cuda_devices(const char *list, int *out){
@@ -195,7 +195,7 @@ static double rss_gb(void){ struct rusage r; getrusage(RUSAGE_SELF,&r);
static float *falloc(int64_t n){ static float *falloc(int64_t n){
/* guardia anti-wrap (report PR #25): n assurdo da file modello ostili non deve /* guardia anti-wrap (report PR #25): n assurdo da file modello ostili non deve
* diventare una malloc piccola. Niente calloc: il memset nel percorso caldo costa. */ * diventare una malloc piccola. Niente calloc: il memset nel percorso caldo costa. */
if(n<0 || (uint64_t)n > SIZE_MAX/sizeof(float)){ fprintf(stderr,"falloc: n=%lld fuori range\n",(long long)n); exit(1); } if(n<0 || (uint64_t)n > SIZE_MAX/sizeof(float)){ fprintf(stderr,"falloc: n=%lld is out of range\n",(long long)n); exit(1); }
float *p=malloc((size_t)n*sizeof(float)); if(!p){fprintf(stderr,"OOM\n");exit(1);} return p; } float *p=malloc((size_t)n*sizeof(float)); if(!p){fprintf(stderr,"OOM\n");exit(1);} return p; }
/* y[S,O] = x[S,I] @ W^T, W[O,I] f32 */ /* y[S,O] = x[S,I] @ W^T, W[O,I] f32 */
@@ -481,7 +481,7 @@ static void matmul_qt(float *y, const float *x, QT *w, int S){
: w->fmt==1 ? (const void*)w->q8 : (const void*)w->q4; : w->fmt==1 ? (const void*)w->q8 : (const void*)w->q4;
if(coli_cuda_matmul(&w->cuda,y,x,weights,w->s,w->fmt,S,w->I,w->O,w->cuda_device)) return; if(coli_cuda_matmul(&w->cuda,y,x,weights,w->s,w->fmt,S,w->I,w->O,w->cuda_device)) return;
w->cuda_failed=1; w->cuda_failed=1;
fprintf(stderr,"[CUDA] tensor [%d,%d] su device %d disabilitato dopo errore; fallback CPU\n", fprintf(stderr,"[CUDA] tensor [%d,%d] on device %d disabled after an error; falling back to CPU\n",
w->O,w->I,w->cuda_device); w->O,w->I,w->cuda_device);
} }
#endif #endif
@@ -681,11 +681,11 @@ static void load_cfg(Cfg *c, const char *snap){
} } } }
c->qk_head=c->qk_nope+c->qk_rope; c->qk_head=c->qk_nope+c->qk_rope;
c->attn_scale = 1.f / sqrtf((float)c->qk_head); c->attn_scale = 1.f / sqrtf((float)c->qk_head);
if(c->n_group!=1){ fprintf(stderr,"questo motore assume n_group=1 (GLM-5.2)\n"); exit(1); } if(c->n_group!=1){ fprintf(stderr,"this engine requires n_group=1 (GLM-5.2)\n"); exit(1); }
/* VALIDAZIONE (report PR #25): il config.json arriva da mirror non fidati — dimensioni /* VALIDAZIONE (report PR #25): il config.json arriva da mirror non fidati — dimensioni
* ostili non devono superare questo punto. Un solo choke point protegge ogni alloc a valle. */ * ostili non devono superare questo punto. Un solo choke point protegge ogni alloc a valle. */
#define CKR(name,v,lo,hi) if((v)<(lo)||(v)>(hi)){ \ #define CKR(name,v,lo,hi) if((v)<(lo)||(v)>(hi)){ \
fprintf(stderr,"config: %s=%d fuori range [%d,%d]\n",name,(int)(v),(int)(lo),(int)(hi)); exit(1); } fprintf(stderr,"config: %s=%d is outside [%d,%d]\n",name,(int)(v),(int)(lo),(int)(hi)); exit(1); }
CKR("hidden_size",c->hidden,1,1<<20) CKR("num_hidden_layers",c->n_layers,1,128) CKR("hidden_size",c->hidden,1,1<<20) CKR("num_hidden_layers",c->n_layers,1,128)
CKR("num_attention_heads",c->n_heads,1,1024) CKR("n_routed_experts",c->n_experts,1,4096) CKR("num_attention_heads",c->n_heads,1,1024) CKR("n_routed_experts",c->n_experts,1,4096)
CKR("num_experts_per_tok",c->topk,1,64) CKR("moe_intermediate_size",c->moe_inter,1,1<<20) CKR("num_experts_per_tok",c->topk,1,64) CKR("moe_intermediate_size",c->moe_inter,1,1<<20)
@@ -729,7 +729,7 @@ static QT qt_load(Model *m, const char *name, int O, int I, int bits){
return t; return t;
} }
static float *ld(Model *m, const char *name){ /* tensore 1D f32 residente (norme/bias) */ static float *ld(Model *m, const char *name){ /* tensore 1D f32 residente (norme/bias) */
int64_t n=st_numel(&m->S,name); if(n<0){fprintf(stderr,"manca %s\n",name);exit(1);} int64_t n=st_numel(&m->S,name); if(n<0){fprintf(stderr,"missing %s\n",name);exit(1);}
float *p=falloc(n); st_read_f32(&m->S,name,p,0); return p; float *p=falloc(n); st_read_f32(&m->S,name,p,0); return p;
} }
@@ -852,7 +852,7 @@ static void model_init(Model *m, const char *snap, int cap, int ebits, int dbits
m->ix_knw[i]=ld(m,PI("k_norm.weight")); m->ix_knb[i]=ld(m,PI("k_norm.bias")); m->ix_knw[i]=ld(m,PI("k_norm.weight")); m->ix_knb[i]=ld(m,PI("k_norm.bias"));
#undef PI #undef PI
} }
fprintf(stderr,"[DSA] indexer attivo: attenzione sparsa top-%d oltre %d token di contesto\n", fprintf(stderr,"[DSA] indexer active: top-%d sparse attention beyond %d context tokens\n",
c->index_topk, c->index_topk); c->index_topk, c->index_topk);
} }
} }
@@ -912,7 +912,7 @@ static void expert_load(Model *m, int layer, int eid, ESlot *s){
for(int k=0;k<3;k++){ for(int k=0;k<3;k++){
tw[k]=st_find(&m->S,nm[k]); tw[k]=st_find(&m->S,nm[k]);
snprintf(qn,sizeof(qn),"%s.qs",nm[k]); tq[k]=st_find(&m->S,qn); snprintf(qn,sizeof(qn),"%s.qs",nm[k]); tq[k]=st_find(&m->S,qn);
if(!tw[k]||!tq[k]){ fprintf(stderr,"manca %s\n",nm[k]); exit(1); } if(!tw[k]||!tq[k]){ fprintf(stderr,"missing %s\n",nm[k]); exit(1); }
} }
int64_t wtot=tw[0]->nbytes+tw[1]->nbytes+tw[2]->nbytes; int64_t wtot=tw[0]->nbytes+tw[1]->nbytes+tw[2]->nbytes;
int64_t ftot=(tq[0]->nbytes+tq[1]->nbytes+tq[2]->nbytes)/4; int64_t ftot=(tq[0]->nbytes+tq[1]->nbytes+tq[2]->nbytes)/4;
@@ -1548,21 +1548,21 @@ static inline int argmax_v(const float *lo, int V){
static void grammar_setup(Tok *T){ static void grammar_setup(Tok *T){
const char *gf=getenv("GRAMMAR"); if(!gf||!*gf) return; const char *gf=getenv("GRAMMAR"); if(!gf||!*gf) return;
FILE *f=fopen(gf,"rb"); FILE *f=fopen(gf,"rb");
if(!f){ fprintf(stderr,"[GRAMMAR] impossibile aprire %s\n",gf); return; } if(!f){ fprintf(stderr,"[GRAMMAR] cannot open %s\n",gf); return; }
fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET); fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET);
char *txt=malloc((size_t)n+1); char *txt=malloc((size_t)n+1);
if(!txt || fread(txt,1,(size_t)n,f)!=(size_t)n){ if(!txt || fread(txt,1,(size_t)n,f)!=(size_t)n){
fprintf(stderr,"[GRAMMAR] lettura fallita: %s\n",gf); fclose(f); free(txt); return; } fprintf(stderr,"[GRAMMAR] failed to read %s\n",gf); fclose(f); free(txt); return; }
fclose(f); txt[n]=0; fclose(f); txt[n]=0;
if(gr_parse(&g_gram,txt)){ fprintf(stderr,"[GRAMMAR] %s: %s\n",gf,g_gram.err); free(txt); return; } if(gr_parse(&g_gram,txt)){ fprintf(stderr,"[GRAMMAR] %s: %s\n",gf,g_gram.err); free(txt); return; }
free(txt); free(txt);
gr_state_init(&g_gst,&g_gram); gr_state_init(&g_gst,&g_gram);
if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammatica non trattabile (ricorsione sinistra?)\n",gf); return; } if(!g_gst.alive){ fprintf(stderr,"[GRAMMAR] %s: grammar cannot be evaluated (left recursion?)\n",gf); return; }
if(getenv("GRAMMAR_DRAFT")) g_gr_max=atoi(getenv("GRAMMAR_DRAFT")); if(getenv("GRAMMAR_DRAFT")) g_gr_max=atoi(getenv("GRAMMAR_DRAFT"));
if(g_gr_max<1) g_gr_max=1; if(g_gr_max<1) g_gr_max=1;
if(g_gr_max>48) g_gr_max=48; if(g_gr_max>48) g_gr_max=48;
g_gr_T=T; g_gr_on=1; g_gr_T=T; g_gr_on=1;
fprintf(stderr,"[GRAMMAR] %s: %d regole, span forzato max %d token/forward\n",gf,g_gram.n,g_gr_max); fprintf(stderr,"[GRAMMAR] %s: %d rules, forced span capped at %d tokens/forward\n",gf,g_gram.n,g_gr_max);
} }
/* stato pulito all'inizio di ogni RISPOSTA (non tra i \x02MORE, che continuano) */ /* stato pulito all'inizio di ogni RISPOSTA (non tra i \x02MORE, che continuano) */
static void grammar_reset(void){ static void grammar_reset(void){
@@ -1592,7 +1592,7 @@ static int grammar_draft(int *draft, int cap){
if(g_gr_prop>=32 && g_gr_acc*2<g_gr_prop){ /* guardia adattiva, come per MTP: if(g_gr_prop>=32 && g_gr_acc*2<g_gr_prop){ /* guardia adattiva, come per MTP:
acceptance sotto il 50% = tokenizzazione fuori asse, meglio spegnersi */ acceptance sotto il 50% = tokenizzazione fuori asse, meglio spegnersi */
g_gr_on=0; g_gr_on=0;
fprintf(stderr,"[GRAMMAR] acceptance %.0f%% dopo %llu proposte: draft grammaticali disattivati\n", fprintf(stderr,"[GRAMMAR] %.0f%% acceptance after %llu proposals: grammar drafts disabled\n",
100.0*g_gr_acc/g_gr_prop,(unsigned long long)g_gr_prop); 100.0*g_gr_acc/g_gr_prop,(unsigned long long)g_gr_prop);
return 0; return 0;
} }
@@ -1655,7 +1655,7 @@ static void stops_arm(const Cfg *c, int tok_eos){
g_nstop=0; g_nstop=0;
for(int i=0;i<c->n_stop;i++) g_stop[g_nstop++]=c->stop_ids[i]; for(int i=0;i<c->n_stop;i++) g_stop[g_nstop++]=c->stop_ids[i];
if(tok_eos>=0 && !is_stop(tok_eos)) g_stop[g_nstop++]=tok_eos; if(tok_eos>=0 && !is_stop(tok_eos)) g_stop[g_nstop++]=tok_eos;
fprintf(stderr,"[stop] %d token di stop:",g_nstop); fprintf(stderr,"[stop] %d stop tokens:",g_nstop);
for(int i=0;i<g_nstop;i++) fprintf(stderr," %d",g_stop[i]); for(int i=0;i<g_nstop;i++) fprintf(stderr," %d",g_stop[i]);
fprintf(stderr,"\n"); fprintf(stderr,"\n");
} }
@@ -1687,7 +1687,7 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
/* auto-off adattivo: draft che non vengono mai accettati = solo tassa disco */ /* auto-off adattivo: draft che non vengono mai accettati = solo tassa disco */
if(m->has_mtp && m->mtp_prop>=24 && m->mtp_acc*10 < m->mtp_prop){ if(m->has_mtp && m->mtp_prop>=24 && m->mtp_acc*10 < m->mtp_prop){
g_draft=0; g_draft=0;
fprintf(stderr,"[MTP] acceptance %.0f%% dopo %llu proposte: draft disattivati\n", fprintf(stderr,"[MTP] %.0f%% acceptance after %llu proposals: drafts disabled\n",
100.0*m->mtp_acc/m->mtp_prop, (unsigned long long)m->mtp_prop); 100.0*m->mtp_acc/m->mtp_prop, (unsigned long long)m->mtp_prop);
} }
} }
@@ -1703,7 +1703,7 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
float *lo=step_all(m,batch,S,kv); m->n_fw++; float *lo=step_all(m,batch,S,kv); m->n_fw++;
int k=0; /* verifica: accetta finche' coincide */ int k=0; /* verifica: accetta finche' coincide */
if(g>0 && getenv("MTP_DEBUG")){ int veri=argmax_v(lo,V); if(g>0 && getenv("MTP_DEBUG")){ int veri=argmax_v(lo,V);
fprintf(stderr,"[mtpdbg] draft0=%d verita=%d %s\n", draft[0], veri, draft[0]==veri?"HIT":"miss"); } fprintf(stderr,"[mtpdbg] draft0=%d verified=%d %s\n", draft[0], veri, draft[0]==veri?"HIT":"miss"); }
while(k<g && emitted<n_new){ while(k<g && emitted<n_new){
int accept; int accept;
if(g_temp<=0) accept = (argmax_v(lo+(int64_t)k*V,V)==draft[k]); if(g_temp<=0) accept = (argmax_v(lo+(int64_t)k*V,V)==draft[k]);
@@ -1808,8 +1808,8 @@ static void generate(Model *m, const int *prompt, int np, int n_new, int *out){
static void profile_print(Model *m, double elapsed){ static void profile_print(Model *m, double elapsed){
double accounted=m->t_edisk+m->t_emm+m->t_attn+m->t_head; double accounted=m->t_edisk+m->t_emm+m->t_attn+m->t_head;
printf("PROFILO: expert-disk %.3fs | expert-matmul %.3fs | attention %.3fs " printf("PROFILE: expert-disk %.3fs | expert-matmul %.3fs | attention %.3fs "
"(di cui kvb %.3fs) | lm_head %.3fs | altro %.3fs\n", "(including kvb %.3fs) | lm_head %.3fs | other %.3fs\n",
m->t_edisk,m->t_emm,m->t_attn,m->t_kvb,m->t_head,elapsed-accounted); m->t_edisk,m->t_emm,m->t_attn,m->t_kvb,m->t_head,elapsed-accounted);
} }
@@ -1817,7 +1817,7 @@ static void profile_print(Model *m, double elapsed){
* replay the oracle sequence one token at a time. CPU and CUDA therefore see * replay the oracle sequence one token at a time. CPU and CUDA therefore see
* identical hidden-state inputs even if their argmax predictions differ. */ * identical hidden-state inputs even if their argmax predictions differ. */
static void run_replay(Model *m, const int *full, int nfull, int np){ static void run_replay(Model *m, const int *full, int nfull, int np){
if(np<2||nfull<=np){ fprintf(stderr,"REPLAY richiede prompt e continuation non vuoti\n"); return; } if(np<2||nfull<=np){ fprintf(stderr,"REPLAY requires a non-empty prompt and continuation\n"); return; }
kv_alloc(m,nfull+2); kv_alloc(m,nfull+2);
float *logit=step(m,full,np-1,0); free(logit); float *logit=step(m,full,np-1,0); free(logit);
m->hits=m->miss=m->ereq=m->gpu_expert_calls=0; m->hits=m->miss=m->ereq=m->gpu_expert_calls=0;
@@ -1827,11 +1827,11 @@ static void run_replay(Model *m, const int *full, int nfull, int np){
logit=step(m,full+i,1,i); free(logit); steps++; logit=step(m,full+i,1,i); free(logit); steps++;
} }
double dt=now_s()-t0, tot=m->hits+m->miss; double dt=now_s()-t0, tot=m->hits+m->miss;
printf("REPLAY decode: %d token in %.3fs | %.2f tok/s | expert hit %.1f%%\n", printf("REPLAY decode: %d tokens in %.3fs | %.2f tok/s | expert hit %.1f%%\n",
steps,dt,steps/dt,tot?100.0*m->hits/tot:0.0); steps,dt,steps/dt,tot?100.0*m->hits/tot:0.0);
profile_print(m,dt); profile_print(m,dt);
#ifdef COLI_CUDA #ifdef COLI_CUDA
if(m->gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n", if(m->gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls); m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print(); if(g_cuda_enabled) cuda_stats_print();
#endif #endif
@@ -1849,8 +1849,8 @@ static void run_text(Model *m, const char *snap, const char *prompt, int ngen){
* distribuzione int4 e' rumore di quantizzazione */ * distribuzione int4 e' rumore di quantizzazione */
int cap=(int)strlen(prompt)+16; int *pids=malloc(cap*sizeof(int)); int cap=(int)strlen(prompt)+16; int *pids=malloc(cap*sizeof(int));
int np=tok_encode(&T,prompt,(int)strlen(prompt),pids,cap); int np=tok_encode(&T,prompt,(int)strlen(prompt),pids,cap);
if(np<1){ fprintf(stderr,"prompt vuoto dopo tokenizzazione\n"); return; } if(np<1){ fprintf(stderr,"prompt is empty after tokenization\n"); return; }
printf("prompt: %d token | genero fino a %d (stop EOS=%d) | draft n-gram=%d\n", np, ngen, eos, g_draft); printf("prompt: %d tokens | generating up to %d (EOS stop=%d) | n-gram draft=%d\n", np, ngen, eos, g_draft);
fputs(prompt,stdout); fflush(stdout); fputs(prompt,stdout); fflush(stdout);
kv_alloc(m, np+ngen+g_draft+2); kv_alloc(m, np+ngen+g_draft+2);
int *all=malloc((np+ngen+g_draft+2)*sizeof(int)); memcpy(all,pids,np*sizeof(int)); int *all=malloc((np+ngen+g_draft+2)*sizeof(int)); memcpy(all,pids,np*sizeof(int));
@@ -1862,24 +1862,24 @@ static void run_text(Model *m, const char *snap, const char *prompt, int ngen){
double dt=now_s()-t; double dt=now_s()-t;
double tot=m->hits+m->miss; double tot=m->hits+m->miss;
int nsp=0; for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) nsp++; int nsp=0; for(int i=0;i<c->n_layers;i++) if(m->L[i].sparse) nsp++;
printf("\n---\n%d token in %.2fs (%.2f tok/s) | hit-rate expert %.1f%% | RSS %.2f GB\n", printf("\n---\n%d tokens in %.2fs (%.2f tok/s) | expert hit rate %.1f%% | RSS %.2f GB\n",
produced, dt, produced/dt, tot?100.0*m->hits/tot:0.0, rss_gb()); produced, dt, produced/dt, tot?100.0*m->hits/tot:0.0, rss_gb());
printf("expert caricati/token: %.1f (per-layer %.2f su %d; baseline topk=%d) | TOPK=%d TOPP=%.2f\n", printf("experts loaded/token: %.1f (per-layer %.2f across %d; baseline topk=%d) | TOPK=%d TOPP=%.2f\n",
produced?(double)m->ereq/produced:0.0, (produced&&nsp)?(double)m->ereq/produced/nsp:0.0, nsp, c->topk, g_topk, g_topp); produced?(double)m->ereq/produced:0.0, (produced&&nsp)?(double)m->ereq/produced/nsp:0.0, nsp, c->topk, g_topk, g_topp);
printf("speculazione: %.2f token/forward (%llu fw per %llu tok) | MTP acceptance %.0f%% (%llu/%llu)\n", printf("speculation: %.2f tokens/forward (%llu forwards per %llu tokens) | MTP acceptance %.0f%% (%llu/%llu)\n",
m->n_fw?(double)m->n_emit/m->n_fw:1.0, (unsigned long long)m->n_fw, (unsigned long long)m->n_emit, m->n_fw?(double)m->n_emit/m->n_fw:1.0, (unsigned long long)m->n_fw, (unsigned long long)m->n_emit,
m->mtp_prop?100.0*m->mtp_acc/m->mtp_prop:0.0, (unsigned long long)m->mtp_acc, (unsigned long long)m->mtp_prop); m->mtp_prop?100.0*m->mtp_acc/m->mtp_prop:0.0, (unsigned long long)m->mtp_acc, (unsigned long long)m->mtp_prop);
if(g_gr_prop) printf("grammatica: acceptance %.0f%% (%llu/%llu draft forzati)\n", if(g_gr_prop) printf("grammar: %.0f%% acceptance (%llu/%llu forced drafts)\n",
100.0*g_gr_acc/g_gr_prop, (unsigned long long)g_gr_acc, (unsigned long long)g_gr_prop); 100.0*g_gr_acc/g_gr_prop, (unsigned long long)g_gr_acc, (unsigned long long)g_gr_prop);
#ifdef COLI_CUDA #ifdef COLI_CUDA
if(m->gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n", if(m->gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls); m->gpu_expert_count,m->gpu_expert_bytes/1e9,(unsigned long long)m->gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print(); if(g_cuda_enabled) cuda_stats_print();
#endif #endif
profile_print(m,dt); profile_print(m,dt);
if(g_looka){ if(g_looka){
const char *nm[3]={"token precedente (=SPEC prefetch)","ingresso layer, salto attention","layer successivo (1 giro di anticipo)"}; const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"};
printf("LOOKAHEAD routing — recall degli expert veri nel top-8 predetto:\n"); printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n");
for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i], for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i],
la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]); la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]);
} }
@@ -1950,11 +1950,11 @@ static void repin_pass(Model *m){
qt_cuda_reset(&s->g); qt_cuda_reset(&s->u); qt_cuda_reset(&s->d); qt_cuda_reset(&s->g); qt_cuda_reset(&s->u); qt_cuda_reset(&s->d);
s->g.cuda_eligible=s->u.cuda_eligible=s->d.cuda_eligible=0; s->g.cuda_eligible=s->u.cuda_eligible=s->d.cuda_eligible=0;
m->gpu_expert_count--; m->gpu_expert_bytes-=old_gpu; m->gpu_expert_count--; m->gpu_expert_bytes-=old_gpu;
fprintf(stderr,"[REPIN] upload VRAM fallito; slot degradato a RAM\n"); fprintf(stderr,"[REPIN] VRAM upload failed; slot downgraded to RAM\n");
} }
} }
#endif #endif
fprintf(stderr,"[REPIN] %s layer %d: esce/out %d (heat=%u) <- entra/in %d (heat=%u) in %.0f ms\n", fprintf(stderr,"[REPIN] %s layer %d: evict %d (heat=%u) <- admit %d (heat=%u) in %.0f ms\n",
tier,cd[b].l,old,old_heat,cd[b].eid,new_heat,(now_s()-t0)*1e3); tier,cd[b].l,old,old_heat,cd[b].eid,new_heat,(now_s()-t0)*1e3);
} }
for(int l=0;l<m->c.n_layers;l++) if(m->eheat[l]) tier_decay(m->eheat[l],m->c.n_experts); for(int l=0;l<m->c.n_layers;l++) if(m->eheat[l]) tier_decay(m->eheat[l],m->c.n_experts);
@@ -2014,11 +2014,11 @@ static int kv_disk_load(Model *m, int *hist, int maxctx){
char mg[8]; int32_t h[8], w[8]; kv_hdr(m,w,0); char mg[8]; int32_t h[8], w[8]; kv_hdr(m,w,0);
if(fread(mg,1,8,f)!=8 || memcmp(mg,KV_MAGIC,8) || fread(h,4,8,f)!=8 || if(fread(mg,1,8,f)!=8 || memcmp(mg,KV_MAGIC,8) || fread(h,4,8,f)!=8 ||
h[0]!=w[0]||h[1]!=w[1]||h[2]!=w[2]||h[3]!=w[3]||h[4]!=w[4]||h[5]!=w[5]){ h[0]!=w[0]||h[1]!=w[1]||h[2]!=w[2]||h[3]!=w[3]||h[4]!=w[4]||h[5]!=w[5]){
fprintf(stderr,"[KV] .coli_kv di un altro modello/versione: ignorato\n"); fclose(f); return 0; } fprintf(stderr,"[KV] ignoring .coli_kv from a different model or version\n"); fclose(f); return 0; }
int nrec=h[6]; int nrec=h[6];
if(nrec<1){ fclose(f); return 0; } if(nrec<1){ fclose(f); return 0; }
if(nrec>=maxctx-8-g_draft){ if(nrec>=maxctx-8-g_draft){
fprintf(stderr,"[KV] conversazione salvata (%d token) piu' grande del contesto: riparto da zero\n",nrec); fprintf(stderr,"[KV] saved conversation (%d tokens) exceeds the context: starting over\n",nrec);
fclose(f); return 0; } fclose(f); return 0; }
double t0=now_s(); double t0=now_s();
for(int p=0;p<nrec;p++){ for(int p=0;p<nrec;p++){
@@ -2034,7 +2034,7 @@ out:
fclose(f); fclose(f);
if(nrec>0){ if(nrec>0){
if(m->has_mtp) m->kv_start[c->n_layers]=-1; /* la finestra MTP riparte da sola */ if(m->has_mtp) m->kv_start[c->n_layers]=-1; /* la finestra MTP riparte da sola */
fprintf(stderr,"[KV] conversazione ripresa dal disco: %d token in %.1fs (niente re-prefill)\n", fprintf(stderr,"[KV] resumed conversation from disk: %d tokens in %.1fs (no re-prefill)\n",
nrec, now_s()-t0); nrec, now_s()-t0);
} }
k->disk_nrec=nrec; k->disk_nrec=nrec;
@@ -2074,12 +2074,12 @@ static void run_serve(Model *m, const char *snap){
int templ=getenv("CHAT_TEMPLATE")?atoi(getenv("CHAT_TEMPLATE")):1; int templ=getenv("CHAT_TEMPLATE")?atoi(getenv("CHAT_TEMPLATE")):1;
g_kvsave = getenv("KVSAVE")?atoi(getenv("KVSAVE")):1; g_kvsave = getenv("KVSAVE")?atoi(getenv("KVSAVE")):1;
int nctx=getenv("KV_SLOTS")?atoi(getenv("KV_SLOTS")):1; int nctx=getenv("KV_SLOTS")?atoi(getenv("KV_SLOTS")):1;
if(nctx<1||nctx>16){ fprintf(stderr,"KV_SLOTS deve essere tra 1 e 16\n"); exit(2); } if(nctx<1||nctx>16){ fprintf(stderr,"KV_SLOTS must be between 1 and 16\n"); exit(2); }
KVState *initial=m->kv; free(initial->kv_start); free(initial); KVState *initial=m->kv; free(initial->kv_start); free(initial);
ServeCtx *ctx=calloc(nctx,sizeof(ServeCtx)); ServeCtx *ctx=calloc(nctx,sizeof(ServeCtx));
for(int i=0;i<nctx;i++) serve_ctx_init(m,&ctx[i],snap,i,maxctx); for(int i=0;i<nctx;i++) serve_ctx_init(m,&ctx[i],snap,i,maxctx);
int active=0; ServeCtx *sc=&ctx[0]; kv_bind(m,&sc->kv); int active=0; ServeCtx *sc=&ctx[0]; kv_bind(m,&sc->kv);
fprintf(stderr,"[KV] context slots: %d x %d token, projected pool %.2f GB\n", fprintf(stderr,"[KV] context slots: %d x %d tokens, projected pool %.2f GB\n",
nctx,maxctx,kv_pool_bytes(m,maxctx)/1e9); nctx,maxctx,kv_pool_bytes(m,maxctx)/1e9);
#define hist (sc->hist) #define hist (sc->hist)
#define len (sc->len) #define len (sc->len)
@@ -2232,7 +2232,7 @@ static void stats_dump_q(Model *m, const char *path, int quiet){
for(int i=0;i<=c->n_layers;i++){ if(!m->eusage[i]) continue; for(int i=0;i<=c->n_layers;i++){ if(!m->eusage[i]) continue;
for(int e=0;e<c->n_experts;e++) if(m->eusage[i][e]){ fprintf(f,"%d %d %u\n",i,e,m->eusage[i][e]); tot+=m->eusage[i][e]; nz++; } } for(int e=0;e<c->n_experts;e++) if(m->eusage[i][e]){ fprintf(f,"%d %d %u\n",i,e,m->eusage[i][e]); tot+=m->eusage[i][e]; nz++; } }
fclose(f); rename(tmp,path); fclose(f); rename(tmp,path);
if(!quiet) fprintf(stderr,"[STATS] %lld selezioni su %lld expert distinti -> %s\n",(long long)tot,(long long)nz,path); if(!quiet) fprintf(stderr,"[STATS] %lld selections across %lld distinct experts -> %s\n",(long long)tot,(long long)nz,path);
} }
static void stats_dump(Model *m, const char *path){ stats_dump_q(m,path,0); } static void stats_dump(Model *m, const char *path){ stats_dump_q(m,path,0); }
@@ -2290,11 +2290,11 @@ static void pin_wire(Model *m){
if(mem_wire(s->fslab, fl)==0) wired+=fl; else failed++; } if(mem_wire(s->fslab, fl)==0) wired+=fl; else failed++; }
} }
if(failed) if(failed)
fprintf(stderr,"[PIN] mlock: %.1f GB inchiodati/wired, %ld alloc fallite/failed " fprintf(stderr,"[PIN] mlock: %.1f GB wired, %ld allocations failed "
"(alza il limite / raise it: ulimit -l unlimited) in %.0fs\n", wired/1e9, failed, now_s()-t0); "(raise the limit: ulimit -l unlimited) in %.0fs\n", wired/1e9, failed, now_s()-t0);
else else
fprintf(stderr,"[PIN] mlock: %.1f GB inchiodati in RAM fisica / wired in physical RAM " fprintf(stderr,"[PIN] mlock: %.1f GB wired in physical RAM "
"(niente compressione/no compression) in %.0fs\n", wired/1e9, now_s()-t0); "(no compression) in %.0fs\n", wired/1e9, now_s()-t0);
} }
static void pin_load(Model *m, const char *statspath, double gb){ static void pin_load(Model *m, const char *statspath, double gb){
@@ -2329,7 +2329,7 @@ static void pin_load(Model *m, const char *statspath, double gb){
expert_load(m,li,r[a].e,&m->pin[li][slot]); expert_load(m,li,r[a].e,&m->pin[li][slot]);
} }
m->resident_bytes += (int64_t)npin*eb; m->resident_bytes += (int64_t)npin*eb;
fprintf(stderr,"[PIN] hot-store: %d expert in RAM (%.1f GB) in %.0fs da %s\n", fprintf(stderr,"[PIN] hot store: %d experts in RAM (%.1f GB) loaded in %.0fs from %s\n",
npin, npin*eb/1e9, now_s()-t0, statspath); npin, npin*eb/1e9, now_s()-t0, statspath);
#ifdef COLI_CUDA #ifdef COLI_CUDA
if(g_cuda_enabled && g_cuda_expert_gb>0){ if(g_cuda_enabled && g_cuda_expert_gb>0){
@@ -2378,9 +2378,9 @@ static void pin_load(Model *m, const char *statspath, double gb){
break; break;
} }
} }
fprintf(stderr,"[CUDA] hot expert tier: %d/%d expert, VRAM %.2f GB (budget totale %.1f GB)\n", fprintf(stderr,"[CUDA] hot expert tier: %d/%d experts, VRAM %.2f GB (total budget %.1f GB)\n",
m->gpu_expert_count,npin,m->gpu_expert_bytes/1e9,g_cuda_expert_gb); m->gpu_expert_count,npin,m->gpu_expert_bytes/1e9,g_cuda_expert_gb);
for(int i=0;i<g_cuda_ndev;i++) fprintf(stderr,"[CUDA] device %d: %d expert, %.2f GB\n", for(int i=0;i<g_cuda_ndev;i++) fprintf(stderr,"[CUDA] device %d: %d experts, %.2f GB\n",
g_cuda_devices[i],placed_n[i],placed_b[i]/1e9); g_cuda_devices[i],placed_n[i],placed_b[i]/1e9);
} }
#endif #endif
@@ -2445,7 +2445,7 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
int auto_b = ram_gb<=0; int auto_b = ram_gb<=0;
if(auto_b){ ram_gb = g_mem_avail_boot*0.88; /* misurata PRIMA del load: il residente gia' if(auto_b){ ram_gb = g_mem_avail_boot*0.88; /* misurata PRIMA del load: il residente gia'
* allocato viene sottratto sotto, non due volte */ * allocato viene sottratto sotto, non due volte */
if(ram_gb<4){ fprintf(stderr,"[RAM] MemAvailable illeggibile/troppo bassa, assumo 8 GB\n"); ram_gb=8; } } if(ram_gb<4){ fprintf(stderr,"[RAM] MemAvailable is unreadable or too low; assuming 8 GB\n"); ram_gb=8; } }
/* slack ONESTO, non forfettario (l'OOM del 2026-07-04 veniva da qui): /* slack ONESTO, non forfettario (l'OOM del 2026-07-04 veniva da qui):
* ws[64] slab del working-set (si materializzano TUTTI nel prefill batch-union), * ws[64] slab del working-set (si materializzano TUTTI nel prefill batch-union),
* KV cache a max_ctx, kvb_all della ricostruzione k/v in attention, * KV cache a max_ctx, kvb_all della ricostruzione k/v in attention,
@@ -2462,8 +2462,8 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
int capmax = (avail>0 && nsp>0) ? (int)(avail/((double)nsp*eb)) : 0; int capmax = (avail>0 && nsp>0) ? (int)(avail/((double)nsp*eb)) : 0;
if(capmax<1) capmax=1; if(capmax<1) capmax=1;
if(capmax < m->ecap){ if(capmax < m->ecap){
fprintf(stderr,"[RAM_GB=%.1f%s] residente %.1f GB + slack %.1f GB (ws %.1f, KV %dx%d %.1f, kvb %.1f), " fprintf(stderr,"[RAM_GB=%.1f%s] resident %.1f GB + reserve %.1f GB (ws %.1f, KV %dx%d %.1f, kvb %.1f), "
"expert %.1f MB x %d layer -> cap abbassato %d->%d (proiezione picco %.1f GB)\n", "experts %.1f MB x %d layers -> cap lowered %d->%d (projected peak %.1f GB)\n",
ram_gb,auto_b?" auto":"",m->resident_bytes/1e9,slack/1e9,ws_b/1e9, ram_gb,auto_b?" auto":"",m->resident_bytes/1e9,slack/1e9,ws_b/1e9,
kv_slot_count(),max_ctx,kv_b/1e9,kvb_b/1e9, kv_slot_count(),max_ctx,kv_b/1e9,kvb_b/1e9,
eb/1e6, nsp, m->ecap, capmax, eb/1e6, nsp, m->ecap, capmax,
@@ -2482,13 +2482,13 @@ static void cap_for_ram(Model *m, double ram_gb, int ebits, int max_ctx){
m->ecache[i]=realloc(m->ecache[i],(size_t)newcap*sizeof(ESlot)); m->ecache[i]=realloc(m->ecache[i],(size_t)newcap*sizeof(ESlot));
memset(m->ecache[i]+m->ecap,0,(size_t)(newcap-m->ecap)*sizeof(ESlot)); memset(m->ecache[i]+m->ecap,0,(size_t)(newcap-m->ecap)*sizeof(ESlot));
} }
fprintf(stderr,"[RAM_GB=%.1f%s] cap ALZATO %d->%d: il budget lo consente " fprintf(stderr,"[RAM_GB=%.1f%s] cap raised %d->%d: budget allows it "
"(proiezione picco %.1f GB; CAP_RAISE=0 per disattivare)\n", "(projected peak %.1f GB; set CAP_RAISE=0 to disable)\n",
ram_gb, auto_b?" auto":"", m->ecap, newcap, ram_gb, auto_b?" auto":"", m->ecap, newcap,
(m->resident_bytes + (double)newcap*nsp*eb + slack)/1e9); (m->resident_bytes + (double)newcap*nsp*eb + slack)/1e9);
m->ecap=newcap; m->ecap=newcap;
} else } else
fprintf(stderr,"[RAM_GB=%.1f%s] cap=%d ok (proiezione picco %.1f GB)\n", ram_gb, auto_b?" auto":"", m->ecap, fprintf(stderr,"[RAM_GB=%.1f%s] cap=%d ok (projected peak %.1f GB)\n", ram_gb, auto_b?" auto":"", m->ecap,
(m->resident_bytes + (double)m->ecap*nsp*eb + slack)/1e9); (m->resident_bytes + (double)m->ecap*nsp*eb + slack)/1e9);
} }
} }
@@ -2523,47 +2523,47 @@ int main(int argc, char **argv){
int ebits= argc>2?atoi(argv[2]):8; int ebits= argc>2?atoi(argv[2]):8;
int dbits= argc>3?atoi(argv[3]):ebits; int dbits= argc>3?atoi(argv[3]):ebits;
if(getenv("SERVE") && (kv_slot_count()<1 || kv_slot_count()>16)){ if(getenv("SERVE") && (kv_slot_count()<1 || kv_slot_count()>16)){
fprintf(stderr,"KV_SLOTS deve essere tra 1 e 16\n"); return 2; fprintf(stderr,"KV_SLOTS must be between 1 and 16\n"); return 2;
} }
#ifdef COLI_CUDA #ifdef COLI_CUDA
if(getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))){ if(getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))){
const char *one=getenv("COLI_GPU"), *many=getenv("COLI_GPUS"); const char *one=getenv("COLI_GPU"), *many=getenv("COLI_GPUS");
if(one&&many){ fprintf(stderr,"usa COLI_GPU oppure COLI_GPUS, non entrambi\n"); return 2; } if(one&&many){ fprintf(stderr,"use COLI_GPU or COLI_GPUS, not both\n"); return 2; }
if(many) g_cuda_ndev=parse_cuda_devices(many,g_cuda_devices); if(many) g_cuda_ndev=parse_cuda_devices(many,g_cuda_devices);
else if(one) g_cuda_ndev=parse_cuda_devices(one,g_cuda_devices); else if(one) g_cuda_ndev=parse_cuda_devices(one,g_cuda_devices);
else { g_cuda_ndev=1; g_cuda_devices[0]=0; } else { g_cuda_ndev=1; g_cuda_devices[0]=0; }
if(g_cuda_ndev<1){ fprintf(stderr,"COLI_GPUS non valido: usa una lista come 0,1,2\n"); return 2; } if(g_cuda_ndev<1){ fprintf(stderr,"invalid COLI_GPUS: use a list such as 0,1,2\n"); return 2; }
g_cuda_enabled=coli_cuda_init(g_cuda_devices,g_cuda_ndev); g_cuda_enabled=coli_cuda_init(g_cuda_devices,g_cuda_ndev);
if(!g_cuda_enabled){ fprintf(stderr,"[CUDA] backend richiesto ma non disponibile\n"); return 2; } if(!g_cuda_enabled){ fprintf(stderr,"[CUDA] requested backend is unavailable\n"); return 2; }
} }
g_cuda_dense=getenv("CUDA_DENSE")?atoi(getenv("CUDA_DENSE")):0; g_cuda_dense=getenv("CUDA_DENSE")?atoi(getenv("CUDA_DENSE")):0;
g_cuda_expert_gb=getenv("CUDA_EXPERT_GB")?atof(getenv("CUDA_EXPERT_GB")):0; g_cuda_expert_gb=getenv("CUDA_EXPERT_GB")?atof(getenv("CUDA_EXPERT_GB")):0;
if((getenv("COLI_GPU")||getenv("COLI_GPUS"))&&!g_cuda_enabled){ fprintf(stderr,"COLI_GPU(S) richiede COLI_CUDA=1\n"); return 2; } if((getenv("COLI_GPU")||getenv("COLI_GPUS"))&&!g_cuda_enabled){ fprintf(stderr,"COLI_GPU(S) requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_dense&&!g_cuda_enabled){ fprintf(stderr,"CUDA_DENSE richiede COLI_CUDA=1\n"); return 2; } if(g_cuda_dense&&!g_cuda_enabled){ fprintf(stderr,"CUDA_DENSE requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_expert_gb>0 && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB richiede COLI_CUDA=1\n"); return 2; } if(g_cuda_expert_gb>0 && !g_cuda_enabled){ fprintf(stderr,"CUDA_EXPERT_GB requires COLI_CUDA=1\n"); return 2; }
if(g_cuda_enabled) fprintf(stderr,"[CUDA] mode: routed experts%s\n",g_cuda_dense?" + resident dense tensors":" only (resident dense on CPU)"); if(g_cuda_enabled) fprintf(stderr,"[CUDA] mode: routed experts%s\n",g_cuda_dense?" + resident dense tensors":" only (resident dense on CPU)");
#else #else
if((getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))) || if((getenv("COLI_CUDA") && atoi(getenv("COLI_CUDA"))) ||
getenv("COLI_GPU") || getenv("COLI_GPUS") || getenv("COLI_GPU") || getenv("COLI_GPUS") ||
(getenv("CUDA_DENSE") && atoi(getenv("CUDA_DENSE"))) || (getenv("CUDA_DENSE") && atoi(getenv("CUDA_DENSE"))) ||
(getenv("CUDA_EXPERT_GB") && atof(getenv("CUDA_EXPERT_GB"))>0)){ (getenv("CUDA_EXPERT_GB") && atof(getenv("CUDA_EXPERT_GB"))>0)){
fprintf(stderr,"CUDA richiesto ma questo binario e' CPU-only; ricompila con: make CUDA=1\n"); fprintf(stderr,"CUDA was requested, but this binary is CPU-only; rebuild with: make CUDA=1\n");
return 2; return 2;
} }
#endif #endif
printf("== Motore C GLM (glm_moe_dsa), cache=%d expert/layer | expert@%d-bit densa@%d-bit | idot: " IDOT_KERNEL " ==\n", cap, ebits, dbits); printf("== GLM C engine (glm_moe_dsa), cache=%d experts/layer | experts@%d-bit dense@%d-bit | idot: " IDOT_KERNEL " ==\n", cap, ebits, dbits);
g_mem_avail_boot = mem_available_gb(); g_mem_avail_boot = mem_available_gb();
Model m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits); Model m; double t0=now_s(); model_init(&m,snap,cap,ebits,dbits);
if(g_draft<0) g_draft = m.has_mtp ? 3 : 0; if(g_draft<0) g_draft = m.has_mtp ? 3 : 0;
if(getenv("DSA_TOPK")) m.c.index_topk=atoi(getenv("DSA_TOPK")); /* override per test */ if(getenv("DSA_TOPK")) m.c.index_topk=atoi(getenv("DSA_TOPK")); /* override per test */
printf("caricato in %.2fs | densa residente: %.2f MB | layers=%d experts=%d | MTP %s (draft=%d)\n", printf("loaded in %.2fs | resident dense: %.2f MB | layers=%d experts=%d | MTP %s (draft=%d)\n",
now_s()-t0, m.resident_bytes/(1024.0*1024.0), m.c.n_layers, m.c.n_experts, now_s()-t0, m.resident_bytes/(1024.0*1024.0), m.c.n_layers, m.c.n_experts,
m.has_mtp?"ATTIVA":"assente", g_draft); m.has_mtp?"ACTIVE":"absent", g_draft);
/* anche su stderr: e' il canale che le UI (coli) mostrano all'utente */ /* anche su stderr: e' il canale che le UI (coli) mostrano all'utente */
fprintf(stderr,"[MTP] %s (draft=%d)\n", m.has_mtp?"attiva: decodifica speculativa nativa":"assente", g_draft); fprintf(stderr,"[MTP] %s (draft=%d)\n", m.has_mtp?"active: native speculative decoding":"absent", g_draft);
if(!strncmp(snap,"/mnt/",5)) if(!strncmp(snap,"/mnt/",5))
fprintf(stderr,"ATTENZIONE: il modello e' su %s (filesystem 9p/Windows, lento e fadvise inefficace).\n" fprintf(stderr,"WARNING: the model is on %s (slow 9p/Windows filesystem; fadvise is ineffective).\n"
" Per RAM e velocita' tienilo su ext4 (es. /home/...).\n", snap); " Keep it on ext4 (for example, /home/...) for memory efficiency and speed.\n", snap);
/* HOT-STORE: PIN=<statsfile> [PIN_GB=g] -> top expert per frequenza fissi in RAM. /* HOT-STORE: PIN=<statsfile> [PIN_GB=g] -> top expert per frequenza fissi in RAM.
* Va PRIMA di cap_for_ram: i pinnati contano nel residente. */ * Va PRIMA di cap_for_ram: i pinnati contano nel residente. */
if(getenv("PIN")) pin_load(&m, getenv("PIN"), getenv("PIN_GB")?atof(getenv("PIN_GB")):10.0); if(getenv("PIN")) pin_load(&m, getenv("PIN"), getenv("PIN_GB")?atof(getenv("PIN_GB")):10.0);
@@ -2574,7 +2574,7 @@ int main(int argc, char **argv){
int est_ctx = getenv("CTX")?atoi(getenv("CTX")):4096; /* stesso default di run_serve */ int est_ctx = getenv("CTX")?atoi(getenv("CTX")):4096; /* stesso default di run_serve */
snprintf(g_usage_path,sizeof(g_usage_path),"%s/.coli_usage",snap); snprintf(g_usage_path,sizeof(g_usage_path),"%s/.coli_usage",snap);
int64_t hist = usage_load(&m,g_usage_path); int64_t hist = usage_load(&m,g_usage_path);
if(hist>0) fprintf(stderr,"[USAGE] storia expert: %lld selezioni (%s)\n",(long long)hist,g_usage_path); if(hist>0) fprintf(stderr,"[USAGE] expert history: %lld selections (%s)\n",(long long)hist,g_usage_path);
int autopin = getenv("AUTOPIN")?atoi(getenv("AUTOPIN")):1; int autopin = getenv("AUTOPIN")?atoi(getenv("AUTOPIN")):1;
if(!getenv("PIN") && autopin && hist>=5000){ if(!getenv("PIN") && autopin && hist>=5000){
/* quota pin proporzionale alla FIDUCIA nella storia: con pochi dati il pin /* quota pin proporzionale alla FIDUCIA nella storia: con pochi dati il pin
@@ -2637,7 +2637,7 @@ int main(int argc, char **argv){
if(pred[i]==tf[i]) ok++; if(pred[i]==tf[i]) ok++;
else fprintf(stderr,"[ORACLE] mismatch pos=%d expected=%d got=%d\n",i,tf[i],pred[i]); else fprintf(stderr,"[ORACLE] mismatch pos=%d expected=%d got=%d\n",i,tf[i],pred[i]);
} }
printf("PREFILL (teacher-forcing) C vs oracolo: %d/%d posizioni | %.1f pos/s\n", printf("PREFILL (teacher-forcing) C vs oracle: %d/%d positions | %.1f pos/s\n",
ok,nfull,nfull/tdt); ok,nfull,nfull/tdt);
if(ok<nfull) fprintf(stderr, if(ok<nfull) fprintf(stderr,
"[ORACLE] %d/%d mismatches — run: TF=1 DEBUG_LOGITS=1 for top-5 logit dump\n", "[ORACLE] %d/%d mismatches — run: TF=1 DEBUG_LOGITS=1 for top-5 logit dump\n",
@@ -2651,23 +2651,23 @@ int main(int argc, char **argv){
int *out=malloc((np+n_new)*sizeof(int)); int *out=malloc((np+n_new)*sizeof(int));
double t=now_s(); generate(&m,prompt,np,n_new,out); double dt=now_s()-t; double t=now_s(); generate(&m,prompt,np,n_new,out); double dt=now_s()-t;
int match=0; int match=0;
printf("\nRiferimento (oracolo): "); for(int i=np;i<nfull;i++) printf("%d ", full[i]); printf("\nReference (oracle): "); for(int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nMotore C GLM : "); for(int i=np;i<nfull;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; } printf("\nGLM C engine : "); for(int i=np;i<nfull;i++){ printf("%d ", out[i]); if(out[i]==full[i])match++; }
printf("\nToken coincidenti: %d/%d\n", match, n_new); printf("\nMatching tokens: %d/%d\n", match, n_new);
double tot=m.hits+m.miss; double tot=m.hits+m.miss;
printf("Speculazione n-gram (DRAFT=%d): %.2f token/forward (%llu fw per %llu tok)\n", printf("N-gram speculation (DRAFT=%d): %.2f tokens/forward (%llu forwards per %llu tokens)\n",
g_draft, m.n_fw?(double)m.n_emit/m.n_fw:1.0, (unsigned long long)m.n_fw, (unsigned long long)m.n_emit); g_draft, m.n_fw?(double)m.n_emit/m.n_fw:1.0, (unsigned long long)m.n_fw, (unsigned long long)m.n_emit);
printf("Hit-rate cache expert: %.1f%% (hit=%llu miss=%llu) | RSS: %.2f GB | %.1f tok/s\n", printf("Expert cache hit rate: %.1f%% (hit=%llu miss=%llu) | RSS: %.2f GB | %.1f tok/s\n",
tot?100.0*m.hits/tot:0.0, (unsigned long long)m.hits, (unsigned long long)m.miss, rss_gb(), n_new/dt); tot?100.0*m.hits/tot:0.0, (unsigned long long)m.hits, (unsigned long long)m.miss, rss_gb(), n_new/dt);
profile_print(&m,dt); profile_print(&m,dt);
#ifdef COLI_CUDA #ifdef COLI_CUDA
if(m.gpu_expert_count) printf("CUDA expert tier: %d residenti (%.2f GB) | %llu chiamate servite da VRAM\n", if(m.gpu_expert_count) printf("CUDA expert tier: %d resident experts (%.2f GB) | %llu calls served from VRAM\n",
m.gpu_expert_count,m.gpu_expert_bytes/1e9,(unsigned long long)m.gpu_expert_calls); m.gpu_expert_count,m.gpu_expert_bytes/1e9,(unsigned long long)m.gpu_expert_calls);
if(g_cuda_enabled) cuda_stats_print(); if(g_cuda_enabled) cuda_stats_print();
#endif #endif
if(g_looka){ if(g_looka){
const char *nm[3]={"token precedente (=SPEC prefetch)","ingresso layer, salto attention","layer successivo (1 giro di anticipo)"}; const char *nm[3]={"previous token (=SPEC prefetch)","layer input, skip attention","next layer (one step ahead)"};
printf("LOOKAHEAD routing — recall degli expert veri nel top-8 predetto:\n"); printf("LOOKAHEAD routing — recall of true experts in predicted top-8:\n");
for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i], for(int i=0;i<3;i++) printf(" %-38s %5.1f%% (%lld/%lld)\n", nm[i],
la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]); la_tot[i]?100.0*la_hit[i]/la_tot[i]:0.0, (long long)la_hit[i], (long long)la_tot[i]);
} }
+25 -25
View File
@@ -124,12 +124,12 @@ static int gr__lit(Grammar *G, int ri, int ai, const char **pp){
while(*p && *p!='"'){ while(*p && *p!='"'){
int b; int b;
if(*p=='\\'){ p++; b=gr__esc(&p); if(*p=='\\'){ p++; b=gr__esc(&p);
if(b<0){ snprintf(G->err,sizeof G->err,"escape non valido nel letterale"); return -1; } } if(b<0){ snprintf(G->err,sizeof G->err,"invalid escape in literal"); return -1; } }
else b=(unsigned char)*p++; else b=(unsigned char)*p++;
GrSym s; memset(&s,0,sizeof s); s.t=GR_CLS; s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7)); GrSym s; memset(&s,0,sizeof s); s.t=GR_CLS; s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7));
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
} }
if(*p!='"'){ snprintf(G->err,sizeof G->err,"letterale non chiuso"); return -1; } if(*p!='"'){ snprintf(G->err,sizeof G->err,"unterminated literal"); return -1; }
*pp=p+1; return 0; *pp=p+1; return 0;
} }
static int gr__cls(Grammar *G, int ri, int ai, const char **pp){ static int gr__cls(Grammar *G, int ri, int ai, const char **pp){
@@ -139,22 +139,22 @@ static int gr__cls(Grammar *G, int ri, int ai, const char **pp){
while(*p && *p!=']'){ while(*p && *p!=']'){
int lo, hi; int lo, hi;
if(*p=='\\'){ p++; lo=gr__esc(&p); if(*p=='\\'){ p++; lo=gr__esc(&p);
if(lo<0){ snprintf(G->err,sizeof G->err,"escape non valido nella classe"); return -1; } } if(lo<0){ snprintf(G->err,sizeof G->err,"invalid escape in character class"); return -1; } }
else lo=(unsigned char)*p++; else lo=(unsigned char)*p++;
hi=lo; hi=lo;
if(*p=='-' && p[1] && p[1]!=']'){ if(*p=='-' && p[1] && p[1]!=']'){
p++; p++;
if(*p=='\\'){ p++; hi=gr__esc(&p); if(*p=='\\'){ p++; hi=gr__esc(&p);
if(hi<0){ snprintf(G->err,sizeof G->err,"escape non valido nella classe"); return -1; } } if(hi<0){ snprintf(G->err,sizeof G->err,"invalid escape in character class"); return -1; } }
else hi=(unsigned char)*p++; else hi=(unsigned char)*p++;
} }
if(hi<lo){ int t=lo; lo=hi; hi=t; } if(hi<lo){ int t=lo; lo=hi; hi=t; }
for(int b=lo;b<=hi;b++) s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7)); for(int b=lo;b<=hi;b++) s.c.bits[b>>3]|=(uint8_t)(1u<<(b&7));
} }
if(*p!=']'){ snprintf(G->err,sizeof G->err,"classe non chiusa"); return -1; } if(*p!=']'){ snprintf(G->err,sizeof G->err,"unterminated character class"); return -1; }
if(neg) for(int i=0;i<32;i++) s.c.bits[i]=(uint8_t)~s.c.bits[i]; if(neg) for(int i=0;i<32;i++) s.c.bits[i]=(uint8_t)~s.c.bits[i];
*pp=p+1; *pp=p+1;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
return 0; return 0;
} }
/* postfisso ? * + sull'ITEM appena letto (simboli [n0, n) dell'alternate corrente). /* postfisso ? * + sull'ITEM appena letto (simboli [n0, n) dell'alternate corrente).
@@ -179,28 +179,28 @@ static int gr__postfix(Grammar *G, int ri, int ai, int n0, char op){
if(gr__push(G,ri,ai,&R)) goto full; /* l'item nell'alternate diventa R */ if(gr__push(G,ri,ai,&R)) goto full; /* l'item nell'alternate diventa R */
return 0; return 0;
full: full:
snprintf(G->err,sizeof G->err,"grammatica troppo grande"); snprintf(G->err,sizeof G->err,"grammar is too large");
return -1; return -1;
} }
static int gr__alts(Grammar *G, int ri, const char **pp, int depth, int in_group){ static int gr__alts(Grammar *G, int ri, const char **pp, int depth, int in_group){
if(depth>32){ snprintf(G->err,sizeof G->err,"gruppi troppo annidati"); return -1; } if(depth>32){ snprintf(G->err,sizeof G->err,"groups are nested too deeply"); return -1; }
const char *p=*pp; const char *p=*pp;
int ai=gr__alt_new(G,ri); int ai=gr__alt_new(G,ri);
if(ai<0){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(ai<0){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
for(;;){ for(;;){
p=gr__ws(p); p=gr__ws(p);
if(!*p){ if(!*p){
if(in_group){ snprintf(G->err,sizeof G->err,"manca ')'"); return -1; } if(in_group){ snprintf(G->err,sizeof G->err,"missing ')'"); return -1; }
break; break;
} }
if(*p==')'){ if(*p==')'){
if(!in_group){ snprintf(G->err,sizeof G->err,"')' inatteso"); return -1; } if(!in_group){ snprintf(G->err,sizeof G->err,"unexpected ')'"); return -1; }
break; break;
} }
if(*p=='|'){ if(*p=='|'){
p++; p++;
ai=gr__alt_new(G,ri); ai=gr__alt_new(G,ri);
if(ai<0){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(ai<0){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
continue; continue;
} }
int n0=G->r[ri].a[ai].n; int n0=G->r[ri].a[ai].n;
@@ -211,24 +211,24 @@ static int gr__alts(Grammar *G, int ri, const char **pp, int depth, int in_group
} else if(*p=='('){ } else if(*p=='('){
p++; p++;
int gi=gr__anon(G); int gi=gr__anon(G);
if(gi<0){ snprintf(G->err,sizeof G->err,"grammatica troppo grande"); return -1; } if(gi<0){ snprintf(G->err,sizeof G->err,"grammar is too large"); return -1; }
if(gr__alts(G,gi,&p,depth+1,1)) return -1; if(gr__alts(G,gi,&p,depth+1,1)) return -1;
p=gr__ws(p); p=gr__ws(p);
if(*p!=')'){ snprintf(G->err,sizeof G->err,"manca ')'"); return -1; } if(*p!=')'){ snprintf(G->err,sizeof G->err,"missing ')'"); return -1; }
p++; p++;
GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)gi; GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)gi;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
} else if(gr__idch(*p)){ } else if(gr__idch(*p)){
int nl=gr__idlen(p); int nl=gr__idlen(p);
const char *after=gr__ws(p+nl); const char *after=gr__ws(p+nl);
if(!in_group && !strncmp(after,"::=",3)) break; /* inizia la prossima regola */ if(!in_group && !strncmp(after,"::=",3)) break; /* inizia la prossima regola */
int ref=gr__rule(G,p,nl); int ref=gr__rule(G,p,nl);
if(ref<0){ snprintf(G->err,sizeof G->err,"troppe regole"); return -1; } if(ref<0){ snprintf(G->err,sizeof G->err,"too many rules"); return -1; }
p+=nl; p+=nl;
GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)ref; GrSym s; memset(&s,0,sizeof s); s.t=GR_REF; s.ref=(int16_t)ref;
if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"memoria esaurita"); return -1; } if(gr__push(G,ri,ai,&s)){ snprintf(G->err,sizeof G->err,"out of memory"); return -1; }
} else { } else {
snprintf(G->err,sizeof G->err,"carattere inatteso '%c'",*p); return -1; snprintf(G->err,sizeof G->err,"unexpected character '%c'",*p); return -1;
} }
p=gr__ws(p); p=gr__ws(p);
if(*p=='?'||*p=='*'||*p=='+'){ if(gr__postfix(G,ri,ai,n0,*p)) return -1; p++; } if(*p=='?'||*p=='*'||*p=='+'){ if(gr__postfix(G,ri,ai,n0,*p)) return -1; p++; }
@@ -244,21 +244,21 @@ static int gr_parse(Grammar *G, const char *src){
p=gr__ws(p); p=gr__ws(p);
if(!*p) break; if(!*p) break;
int nl=gr__idlen(p); int nl=gr__idlen(p);
if(nl<=0){ snprintf(G->err,sizeof G->err,"attesa una regola, trovato '%c'",*p); return -1; } if(nl<=0){ snprintf(G->err,sizeof G->err,"expected a rule, found '%c'",*p); return -1; }
const char *name=p; const char *name=p;
const char *q=gr__ws(p+nl); const char *q=gr__ws(p+nl);
if(strncmp(q,"::=",3)){ snprintf(G->err,sizeof G->err,"atteso '::=' dopo '%.*s'",nl,name); return -1; } if(strncmp(q,"::=",3)){ snprintf(G->err,sizeof G->err,"expected '::=' after '%.*s'",nl,name); return -1; }
p=q+3; p=q+3;
int ri=gr__rule(G,name,nl); int ri=gr__rule(G,name,nl);
if(ri<0){ snprintf(G->err,sizeof G->err,"troppe regole"); return -1; } if(ri<0){ snprintf(G->err,sizeof G->err,"too many rules"); return -1; }
if(G->r[ri].n>0){ snprintf(G->err,sizeof G->err,"regola '%.*s' duplicata",nl,name); return -1; } if(G->r[ri].n>0){ snprintf(G->err,sizeof G->err,"duplicate rule '%.*s'",nl,name); return -1; }
if(gr__alts(G,ri,&p,0,0)) return -1; if(gr__alts(G,ri,&p,0,0)) return -1;
} }
for(int i=0;i<G->n;i++){ for(int i=0;i<G->n;i++){
if(!strcmp(G->r[i].name,"root")) G->root=i; if(!strcmp(G->r[i].name,"root")) G->root=i;
if(G->r[i].n==0){ snprintf(G->err,sizeof G->err,"regola '%s' usata ma mai definita",G->r[i].name); return -1; } if(G->r[i].n==0){ snprintf(G->err,sizeof G->err,"rule '%s' is used but never defined",G->r[i].name); return -1; }
} }
if(G->root<0){ snprintf(G->err,sizeof G->err,"manca la regola 'root'"); return -1; } if(G->root<0){ snprintf(G->err,sizeof G->err,"missing 'root' rule"); return -1; }
return 0; return 0;
} }
static void gr_free(Grammar *G){ static void gr_free(Grammar *G){
+5 -5
View File
@@ -18,26 +18,26 @@
#endif #endif
static double now(){ struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t); return t.tv_sec+t.tv_nsec*1e-9; } static double now(){ struct timespec t; clock_gettime(CLOCK_MONOTONIC,&t); return t.tv_sec+t.tv_nsec*1e-9; }
int main(int argc,char**argv){ int main(int argc,char**argv){
if(argc<2){fprintf(stderr,"uso: %s file [blkMB] [n] [threads] [direct 0/1]\n",argv[0]);return 1;} if(argc<2){fprintf(stderr,"usage: %s file [blkMB] [n] [threads] [direct 0/1]\n",argv[0]);return 1;}
long blk=(argc>2?atol(argv[2]):19)*1024*1024; long blk=(argc>2?atol(argv[2]):19)*1024*1024;
int n=argc>3?atoi(argv[3]):64; int n=argc>3?atoi(argv[3]):64;
int nth=argc>4?atoi(argv[4]):8; int nth=argc>4?atoi(argv[4]):8;
int direct=argc>5?atoi(argv[5]):1; int direct=argc>5?atoi(argv[5]):1;
#ifdef O_DIRECT #ifdef O_DIRECT
int fd=open(argv[1],O_RDONLY|(direct?O_DIRECT:0)); int fd=open(argv[1],O_RDONLY|(direct?O_DIRECT:0));
if(fd<0 && direct){ fprintf(stderr,"O_DIRECT non disponibile (%s), uso buffered\n",strerror(errno)); if(fd<0 && direct){ fprintf(stderr,"O_DIRECT is unavailable (%s); using buffered I/O\n",strerror(errno));
direct=0; fd=open(argv[1],O_RDONLY); } direct=0; fd=open(argv[1],O_RDONLY); }
#else #else
int fd=open(argv[1],O_RDONLY); /* macOS: F_NOCACHE ~ O_DIRECT */ int fd=open(argv[1],O_RDONLY); /* macOS: F_NOCACHE ~ O_DIRECT */
#ifdef __APPLE__ #ifdef __APPLE__
if(direct && fd>=0) fcntl(fd,F_NOCACHE,1); if(direct && fd>=0) fcntl(fd,F_NOCACHE,1);
#else #else
if(direct){ fprintf(stderr,"O_DIRECT non disponibile, uso buffered\n"); direct=0; } if(direct){ fprintf(stderr,"O_DIRECT is unavailable; using buffered I/O\n"); direct=0; }
#endif #endif
#endif #endif
if(fd<0){perror("open");return 1;} if(fd<0){perror("open");return 1;}
off_t sz=lseek(fd,0,SEEK_END); off_t sz=lseek(fd,0,SEEK_END);
if(sz<blk*2){fprintf(stderr,"file troppo piccolo\n");return 1;} if(sz<blk*2){fprintf(stderr,"file is too small\n");return 1;}
/* offset random pre-generati (stessi per ogni configurazione: srand fisso). /* offset random pre-generati (stessi per ogni configurazione: srand fisso).
* 30 bit di rand combinati: su Windows RAND_MAX=32767 e un singolo rand()*4096 * 30 bit di rand combinati: su Windows RAND_MAX=32767 e un singolo rand()*4096
* copre solo i primi 134 MB del file (tutti in page cache = misura falsa). */ * copre solo i primi 134 MB del file (tutti in page cache = misura falsa). */
@@ -55,7 +55,7 @@ int main(int argc,char**argv){
compat_aligned_free(buf); /* su Windows posix_memalign=_aligned_malloc: free() corrompe l'heap */ compat_aligned_free(buf); /* su Windows posix_memalign=_aligned_malloc: free() corrompe l'heap */
} }
double dt=now()-t0; double dt=now()-t0;
printf("%s x%d thread: %d letture x %ldMB = %.1f GB in %.2fs -> %.2f GB/s (%.1f ms/blocco effettivi)\n", printf("%s x%d threads: %d reads x %ldMB = %.1f GB in %.2fs -> %.2f GB/s (%.1f effective ms/block)\n",
direct?"O_DIRECT":"buffered", nth, n, blk/1024/1024, tot/1e9, dt, tot/1e9/dt, dt/n*1000); direct?"O_DIRECT":"buffered", nth, n, blk/1024/1024, tot/1e9, dt, tot/1e9/dt, dt/n*1000);
close(fd); free(offs); return 0; close(fd); free(offs); return 0;
} }
+10 -10
View File
@@ -137,7 +137,7 @@ static void load_cfg(Cfg *c, const char *snap) {
static float *load_t(Model *m, const char *name) { static float *load_t(Model *m, const char *name) {
int64_t n = st_numel(&m->S, name); int64_t n = st_numel(&m->S, name);
if (n < 0) { fprintf(stderr, "manca %s\n", name); exit(1); } if (n < 0) { fprintf(stderr, "missing %s\n", name); exit(1); }
float *p = falloc(n); float *p = falloc(n);
st_read_f32(&m->S, name, p, 0); /* densa: niente DONTNEED, resta residente */ st_read_f32(&m->S, name, p, 0); /* densa: niente DONTNEED, resta residente */
return p; return p;
@@ -357,7 +357,7 @@ static int *read_int_array(jval *o, const char *key, int *n_out) {
int main(int argc, char **argv) { int main(int argc, char **argv) {
const char *snap = getenv("SNAP"); const char *snap = getenv("SNAP");
if (!snap) { fprintf(stderr, "imposta SNAP=<dir snapshot>\n"); return 1; } if (!snap) { fprintf(stderr, "set SNAP=<snapshot directory>\n"); return 1; }
int cap = argc > 1 ? atoi(argv[1]) : 16; int cap = argc > 1 ? atoi(argv[1]) : 16;
int bits = argc > 2 ? atoi(argv[2]) : 8; int bits = argc > 2 ? atoi(argv[2]) : 8;
const char *refpath = argc > 3 ? argv[3] : "ref.json"; const char *refpath = argc > 3 ? argv[3] : "ref.json";
@@ -369,9 +369,9 @@ int main(int argc, char **argv) {
int np, nfull; int *prompt = read_int_array(ref,"prompt_ids",&np); int *full = read_int_array(ref,"full_ids",&nfull); int np, nfull; int *prompt = read_int_array(ref,"prompt_ids",&np); int *full = read_int_array(ref,"full_ids",&nfull);
int n_new = nfull - np; int n_new = nfull - np;
printf("== Motore C streaming, cache = %d expert/layer, expert @ %d-bit ==\n", cap, bits); printf("== Streaming C engine, cache = %d experts/layer, experts @ %d-bit ==\n", cap, bits);
Model m; model_init(&m, snap, cap, bits); Model m; model_init(&m, snap, cap, bits);
printf("densa caricata in %.1fs | RSS dopo load densa: %.2f GB\n", m.dense_load_s, rss_gb()); printf("resident weights loaded in %.1fs | RSS after load: %.2f GB\n", m.dense_load_s, rss_gb());
int *out = malloc((np + n_new) * sizeof(int)); int *out = malloc((np + n_new) * sizeof(int));
double t = now_s(); double t = now_s();
@@ -379,14 +379,14 @@ int main(int argc, char **argv) {
double dt = now_s() - t; double dt = now_s() - t;
int match = 0; int match = 0;
printf("\nRiferimento: "); for (int i=np;i<nfull;i++) printf("%d ", full[i]); printf("\nReference: "); for (int i=np;i<nfull;i++) printf("%d ", full[i]);
printf("\nMotore C : "); for (int i=np;i<nfull;i++) { printf("%d ", out[i]); if (out[i]==full[i]) match++; } printf("\nC engine : "); for (int i=np;i<nfull;i++) { printf("%d ", out[i]); if (out[i]==full[i]) match++; }
printf("\nToken coincidenti: %d/%d\n", match, n_new); printf("\nMatching tokens: %d/%d\n", match, n_new);
double tot = m.hits + m.miss; double tot = m.hits + m.miss;
printf("\nRSS PICCO: %.2f GB\n", rss_gb()); printf("\nPEAK RSS: %.2f GB\n", rss_gb());
printf("Hit-rate cache expert: %.1f%% (hit=%llu miss=%llu)\n", tot?100.0*m.hits/tot:0.0, printf("Expert cache hit rate: %.1f%% (hit=%llu miss=%llu)\n", tot?100.0*m.hits/tot:0.0,
(unsigned long long)m.hits, (unsigned long long)m.miss); (unsigned long long)m.hits, (unsigned long long)m.miss);
printf("Velocita': %.2f tok/s (%.1fs per %d token)\n", n_new/dt, dt, n_new); printf("Speed: %.2f tok/s (%.1fs for %d tokens)\n", n_new/dt, dt, n_new);
free(buf); free(arena); free(buf); free(arena);
return 0; return 0;
} }
+7 -7
View File
@@ -15,24 +15,24 @@ NGEN="${2:-64}"
cd "$CODE" cd "$CODE"
# 0) sanity: il modello deve stare su ext4, non su 9p/Windows # 0) sanity: il modello deve stare su ext4, non su 9p/Windows
case "$DIR" in /mnt/*) echo "ERRORE: $DIR e' su /mnt (9p/Windows). Mettilo su ext4."; exit 1;; esac case "$DIR" in /mnt/*) echo "ERROR: $DIR is on /mnt (9p/Windows). Move it to ext4."; exit 1;; esac
# 1) se un rsync di spostamento e' ancora vivo, aspettalo # 1) se un rsync di spostamento e' ancora vivo, aspettalo
while pgrep -f "rsync.*glm52_i4" >/dev/null 2>&1; do while pgrep -f "rsync.*glm52_i4" >/dev/null 2>&1; do
echo "[1/4] attendo lo spostamento su ext4... ($(du -sh "$DIR" 2>/dev/null | cut -f1))"; sleep 20 echo "[1/4] waiting for the move to ext4... ($(du -sh "$DIR" 2>/dev/null | cut -f1))"; sleep 20
done done
echo "[1/4] spostamento completato: $(du -sh "$DIR" | cut -f1), shard $(ls "$DIR"/*.safetensors 2>/dev/null | wc -l)" echo "[1/4] move complete: $(du -sh "$DIR" | cut -f1), shards $(ls "$DIR"/*.safetensors 2>/dev/null | wc -l)"
# 2) riprende+completa la conversione (ripartibile: salta gli shard gia' fatti) # 2) riprende+completa la conversione (ripartibile: salta gli shard gia' fatti)
echo "[2/4] conversione (riprende da dove era): output -> $DIR" echo "[2/4] conversion (resumes where it stopped): output -> $DIR"
python3 tools/convert_fp8_to_int4.py --repo "$REPO" --outdir "$DIR" --ebits 4 --io-bits 8 python3 tools/convert_fp8_to_int4.py --repo "$REPO" --outdir "$DIR" --ebits 4 --io-bits 8
# 3) il motore richiede tokenizer.json + config.json nella dir del modello # 3) il motore richiede tokenizer.json + config.json nella dir del modello
for f in config.json tokenizer.json; do for f in config.json tokenizer.json; do
[ -f "$DIR/$f" ] || { echo "ERRORE: manca $DIR/$f"; exit 1; } [ -f "$DIR/$f" ] || { echo "ERROR: missing $DIR/$f"; exit 1; }
done done
echo "[3/4] compilo il motore"; make -s glm echo "[3/4] building the engine"; make -s glm
# 4) generazione reale, con auto-cap dal budget RAM e heartbeat RSS su stderr # 4) generazione reale, con auto-cap dal budget RAM e heartbeat RSS su stderr
echo "[4/4] genero (RAM_GB=$RAM_GB, NGEN=$NGEN)"; echo "------" echo "[4/4] generating (RAM_GB=$RAM_GB, NGEN=$NGEN)"; echo "------"
SNAP="$DIR" RAM_GB="$RAM_GB" PROMPT="$PROMPT" NGEN="$NGEN" ./glm 64 SNAP="$DIR" RAM_GB="$RAM_GB" PROMPT="$PROMPT" NGEN="$NGEN" ./glm 64
+6 -6
View File
@@ -13,7 +13,7 @@ STALL_S=180 # secondi senza crescita del download -> riavvio
CONVLOG=/tmp/convert_supervised.log CONVLOG=/tmp/convert_supervised.log
exec 9>"$DIR/.supervisor.lock" exec 9>"$DIR/.supervisor.lock"
flock -n 9 || { echo "supervisore gia' attivo, esco"; exit 1; } flock -n 9 || { echo "a supervisor is already running; exiting"; exit 1; }
log(){ echo "[$(date +%H:%M:%S)] $*"; } log(){ echo "[$(date +%H:%M:%S)] $*"; }
@@ -21,16 +21,16 @@ start_conv(){
cd "$CODE" cd "$CODE"
nohup python3 tools/convert_fp8_to_int4.py --repo zai-org/GLM-5.2-FP8 \ nohup python3 tools/convert_fp8_to_int4.py --repo zai-org/GLM-5.2-FP8 \
--outdir "$DIR" --ebits 4 --io-bits 8 >> "$CONVLOG" 2>&1 & --outdir "$DIR" --ebits 4 --io-bits 8 >> "$CONVLOG" 2>&1 &
log "convertitore avviato (PID $!)" log "converter started (PID $!)"
} }
last_size=-1; stall=0 last_size=-1; stall=0
while :; do while :; do
done_n=$(ls "$DIR"/out-*.safetensors 2>/dev/null | wc -l) done_n=$(ls "$DIR"/out-*.safetensors 2>/dev/null | wc -l)
if [ "$done_n" -ge "$TOTAL" ]; then log "FATTO: $done_n/$TOTAL shard. Esco."; pkill -f convert_fp8 2>/dev/null; exit 0; fi if [ "$done_n" -ge "$TOTAL" ]; then log "DONE: $done_n/$TOTAL shards. Exiting."; pkill -f convert_fp8 2>/dev/null; exit 0; fi
if ! pgrep -f convert_fp8 >/dev/null; then if ! pgrep -f convert_fp8 >/dev/null; then
log "convertitore non attivo ($done_n/$TOTAL): lo avvio" log "converter is not running ($done_n/$TOTAL): starting it"
start_conv; last_size=-1; stall=0; sleep 20; continue start_conv; last_size=-1; stall=0; sleep 20; continue
fi fi
@@ -40,12 +40,12 @@ while :; do
if [ "$size" = "$last_size" ]; then if [ "$size" = "$last_size" ]; then
stall=$((stall+30)) stall=$((stall+30))
if [ "$stall" -ge "$STALL_S" ]; then if [ "$stall" -ge "$STALL_S" ]; then
log "download FERMO da ${stall}s a $((size/1000000)) MB ($done_n/$TOTAL): riavvio il convertitore" log "download stalled for ${stall}s at $((size/1000000)) MB ($done_n/$TOTAL): restarting the converter"
pkill -f convert_fp8; sleep 5 pkill -f convert_fp8; sleep 5
start_conv; last_size=-1; stall=0 start_conv; last_size=-1; stall=0
fi fi
else else
[ "$last_size" -ge 0 ] && [ "$stall" -ge 60 ] && log "download ripreso ($((size/1000000)) MB)" [ "$last_size" -ge 0 ] && [ "$stall" -ge 60 ] && log "download resumed ($((size/1000000)) MB)"
last_size=$size; stall=0 last_size=$size; stall=0
fi fi
else else
+17 -17
View File
@@ -9,35 +9,35 @@ echo "🐦 colibrì — setup"
UNAME_S=$(uname -s) UNAME_S=$(uname -s)
# 1) dipendenze # 1) dipendenze
command -v make >/dev/null || { echo "manca make"; exit 1; } command -v make >/dev/null || { echo "make is missing"; exit 1; }
case "$UNAME_S" in case "$UNAME_S" in
Darwin) Darwin)
command -v clang >/dev/null || { echo "manca clang (xcode-select --install)"; exit 1; } command -v clang >/dev/null || { echo "clang is missing (run: xcode-select --install)"; exit 1; }
echo " clang: $(clang --version | head -1) · $(sysctl -n hw.ncpu) core" echo " clang: $(clang --version | head -1) · $(sysctl -n hw.ncpu) core"
echo -n " OpenMP: " echo -n " OpenMP: "
if [ -f "$(brew --prefix libomp 2>/dev/null)/lib/libomp.dylib" ]; then echo "ok (libomp)" if [ -f "$(brew --prefix libomp 2>/dev/null)/lib/libomp.dylib" ]; then echo "ok (libomp)"
else echo "libomp assente -> build single-thread (consigliato: brew install libomp)"; fi else echo "libomp is missing -> single-threaded build (recommended: brew install libomp)"; fi
;; ;;
MINGW*|MSYS*) MINGW*|MSYS*)
command -v gcc >/dev/null || { echo "manca gcc (MinGW-w64). Installa: pacman -S mingw-w64-x86_64-gcc make"; exit 1; } command -v gcc >/dev/null || { echo "gcc is missing (MinGW-w64). Install: pacman -S mingw-w64-x86_64-gcc make"; exit 1; }
echo " gcc: $(gcc -dumpversion) · MinGW-w64" echo " gcc: $(gcc -dumpversion) · MinGW-w64"
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "manca libgomp (pacman -S mingw-w64-x86_64-gcc)"; exit 1; } echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "libgomp is missing (pacman -S mingw-w64-x86_64-gcc)"; exit 1; }
;; ;;
*) *)
command -v gcc >/dev/null || { echo "manca gcc (es: sudo apt install build-essential)"; exit 1; } command -v gcc >/dev/null || { echo "gcc is missing (for example: sudo apt install build-essential)"; exit 1; }
echo " gcc: $(gcc -dumpversion) · $(nproc) core" echo " gcc: $(gcc -dumpversion) · $(nproc) core"
echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "manca (libgomp)"; exit 1; } echo -n " OpenMP: "; echo 'int main(){return 0;}' | gcc -fopenmp -xc - -o /tmp/_omp 2>/dev/null && echo ok || { echo "libgomp is missing"; exit 1; }
;; ;;
esac esac
# 2) build: nativa (veloce, per QUESTA macchina). Per un binario da distribuire: make portable # 2) build: nativa (veloce, per QUESTA macchina). Per un binario da distribuire: make portable
echo " compilo (ARCH=${ARCH:-native})…" echo " building (ARCH=${ARCH:-native})…"
make -s glm ARCH="${ARCH:-native}" make -s glm ARCH="${ARCH:-native}"
# 3) self-test sull'oracolo tiny, se presente # 3) self-test sull'oracolo tiny, se presente
if [ -d glm_tiny ] && [ -f ref_glm.json ]; then if [ -d glm_tiny ] && [ -f ref_glm.json ]; then
r=$(SNAP=./glm_tiny TF=1 ./glm 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ posizioni" || true) r=$(SNAP=./glm_tiny TF=1 ./glm 64 16 16 2>/dev/null | grep -oE "[0-9]+/[0-9]+ positions" || true)
echo " self-test motore: ${r:-?} (atteso 32/32)" echo " engine self-test: ${r:-?} (expected 32/32)"
fi fi
# 4) info macchina (la velocità dipende da QUESTI due numeri, non dalla GPU) # 4) info macchina (la velocità dipende da QUESTI due numeri, non dalla GPU)
@@ -53,12 +53,12 @@ MINGW*|MSYS*)
ram=$(awk '/MemTotal/{printf "%.0f", $2/1e6}' /proc/meminfo 2>/dev/null || echo "?") ram=$(awk '/MemTotal/{printf "%.0f", $2/1e6}' /proc/meminfo 2>/dev/null || echo "?")
;; ;;
esac esac
echo " RAM: ${ram} GB (più RAM = più expert in cache = più veloce)" echo " RAM: ${ram} GB (more RAM = more cached experts = faster inference)"
echo echo
echo "pronto. Prossimi passi:" echo "ready. Next steps:"
echo " ./coli build # (gia' fatto)" echo " ./coli build # already done"
echo " ./coli convert --model /percorso/NVMe/glm52_i4 # genera il modello int4 (ore)" echo " ./coli convert --model /path/on/NVMe/glm52_i4 # generate the int4 model (hours)"
echo " ./coli info --model /percorso/NVMe/glm52_i4" echo " ./coli info --model /path/on/NVMe/glm52_i4"
echo " ./coli chat --model /percorso/NVMe/glm52_i4 --ram <GB>" echo " ./coli chat --model /path/on/NVMe/glm52_i4 --ram <GB>"
echo echo
echo "IMPORTANTE: tieni il modello su disco VELOCE (NVMe/ext4), MAI su /mnt/c o rete." echo "IMPORTANT: keep the model on fast storage (NVMe/ext4), never on /mnt/c or a network mount."
+5 -5
View File
@@ -52,7 +52,7 @@ static int st_dtype_code(const char *s) {
if (!strcmp(s, "F32")) return 2; if (!strcmp(s, "F32")) return 2;
if (!strcmp(s, "U8")) return 3; /* dati quantizzati (int4 packed / int8) */ if (!strcmp(s, "U8")) return 3; /* dati quantizzati (int4 packed / int8) */
if (!strcmp(s, "I8")) return 3; if (!strcmp(s, "I8")) return 3;
fprintf(stderr, "dtype non gestito: %s\n", s); exit(1); fprintf(stderr, "unsupported dtype: %s\n", s); exit(1);
} }
static inline float bf16_to_f32(uint16_t h) { static inline float bf16_to_f32(uint16_t h) {
@@ -108,7 +108,7 @@ static void st_init(shards *S, const char *snap_dir) {
while ((e = readdir(d))) { while ((e = readdir(d))) {
const char *dot = strrchr(e->d_name, '.'); const char *dot = strrchr(e->d_name, '.');
if (dot && !strcmp(dot, ".safetensors")) { /* model.safetensors o model-0000N-of-... */ if (dot && !strcmp(dot, ".safetensors")) { /* model.safetensors o model-0000N-of-... */
if (nf >= ST_MAX_SHARDS) { fprintf(stderr, "troppi shard (>%d): alza ST_MAX_SHARDS\n", ST_MAX_SHARDS); exit(1); } if (nf >= ST_MAX_SHARDS) { fprintf(stderr, "too many shards (>%d): raise ST_MAX_SHARDS\n", ST_MAX_SHARDS); exit(1); }
snprintf(files[nf++], 1024, "%s/%s", snap_dir, e->d_name); snprintf(files[nf++], 1024, "%s/%s", snap_dir, e->d_name);
} }
} }
@@ -182,7 +182,7 @@ static void st_prefetch(shards *S, const char *name) {
* drop=1 -> consiglia al kernel di scartare le pagine (per gli expert in streaming). */ * drop=1 -> consiglia al kernel di scartare le pagine (per gli expert in streaming). */
static int64_t st_read_f32(shards *S, const char *name, float *out, int drop) { static int64_t st_read_f32(shards *S, const char *name, float *out, int drop) {
st_tensor *t = st_find(S, name); st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); } if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
void *raw = malloc(t->nbytes); void *raw = malloc(t->nbytes);
if (pread(t->fd, raw, t->nbytes, t->off) != t->nbytes) { perror("pread data"); exit(1); } if (pread(t->fd, raw, t->nbytes, t->off) != t->nbytes) { perror("pread data"); exit(1); }
if (t->dtype == 2) { if (t->dtype == 2) {
@@ -208,7 +208,7 @@ static int64_t st_nbytes(shards *S, const char *name) {
* quantizzati int4/int8 del nostro container (dtype U8). drop=1 -> fadvise DONTNEED. */ * quantizzati int4/int8 del nostro container (dtype U8). drop=1 -> fadvise DONTNEED. */
static void st_read_raw(shards *S, const char *name, void *out, int drop) { static void st_read_raw(shards *S, const char *name, void *out, int drop) {
st_tensor *t = st_find(S, name); st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); } if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
if (pread(t->fd, out, t->nbytes, t->off) != t->nbytes) { perror("pread raw"); exit(1); } if (pread(t->fd, out, t->nbytes, t->off) != t->nbytes) { perror("pread raw"); exit(1); }
if (drop) posix_fadvise(t->fd, t->off, t->nbytes, POSIX_FADV_DONTNEED); if (drop) posix_fadvise(t->fd, t->off, t->nbytes, POSIX_FADV_DONTNEED);
} }
@@ -218,7 +218,7 @@ static void st_read_raw(shards *S, const char *name, void *out, int drop) {
* solo expert richiesto via pread del sotto-range, niente lettura dell'intero blocco. */ * solo expert richiesto via pread del sotto-range, niente lettura dell'intero blocco. */
static void st_read_slice_f32(shards *S, const char *name, int64_t elem_off, int64_t n_elems, float *out, int drop) { static void st_read_slice_f32(shards *S, const char *name, int64_t elem_off, int64_t n_elems, float *out, int drop) {
st_tensor *t = st_find(S, name); st_tensor *t = st_find(S, name);
if (!t) { fprintf(stderr, "tensore mancante: %s\n", name); exit(1); } if (!t) { fprintf(stderr, "missing tensor: %s\n", name); exit(1); }
int esz = (t->dtype == 2) ? 4 : 2; int esz = (t->dtype == 2) ? 4 : 2;
int64_t boff = t->off + elem_off * esz, nb = n_elems * esz; int64_t boff = t->off + elem_off * esz, nb = n_elems * esz;
void *raw = malloc(nb); void *raw = malloc(nb);
+1 -1
View File
@@ -5,7 +5,7 @@ from tools.benchmark_cuda_fixture import parse_output
SAMPLE = """ SAMPLE = """
REPLAY decode: 4 tokens | 12.34 tok/s REPLAY decode: 4 tokens | 12.34 tok/s
PROFILO: expert-disk 1.25s | expert-matmul 2.50s | attention 0.75s | lm_head 0.10s | altro -0.05s PROFILE: expert-disk 1.25s | expert-matmul 2.50s | attention 0.75s | lm_head 0.10s | other -0.05s
""" """
+49
View File
@@ -0,0 +1,49 @@
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
HERE = Path(__file__).resolve().parent.parent
CLI = HERE / "coli"
class CliOutputLanguageTest(unittest.TestCase):
def run_cli(self, *args):
return subprocess.run(
[sys.executable, str(CLI), *args],
cwd=HERE,
text=True,
capture_output=True,
check=False,
timeout=10,
)
def test_help_is_english(self):
result = self.run_cli("--help")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("run GLM-5.2 locally", result.stdout)
self.assertIn("automatically apply the RAM/VRAM plan", result.stdout)
self.assertNotIn("modello", result.stdout.lower())
self.assertNotIn("motore", result.stdout.lower())
def test_info_status_is_english(self):
with tempfile.TemporaryDirectory() as model:
result = self.run_cli("info", "--model", model)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("config.json is missing", result.stdout)
self.assertIn("disk", result.stdout)
self.assertIn("engine", result.stdout)
def test_missing_model_error_is_english(self):
with tempfile.TemporaryDirectory() as directory:
missing_model = str(Path(directory) / "missing-model")
result = self.run_cli("run", "--model", missing_model, "hello")
self.assertNotEqual(result.returncode, 0)
self.assertIn("model not found", result.stderr)
self.assertIn("set COLI_MODEL or use --model", result.stderr)
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -5,10 +5,10 @@
#include "../tok.h" #include "../tok.h"
int main(int argc, char **argv){ int main(int argc, char **argv){
if(argc<2){ fprintf(stderr,"uso: %s tokenizer.json < casi\n",argv[0]); return 1; } if(argc<2){ fprintf(stderr,"usage: %s tokenizer.json < cases\n",argv[0]); return 1; }
Tok T; Tok T;
tok_load(&T, argv[1]); tok_load(&T, argv[1]);
fprintf(stderr,"caricato: vocab_ids=%d specials=%d\n", T.n_ids, T.nsp); fprintf(stderr,"loaded: vocab_ids=%d specials=%d\n", T.n_ids, T.nsp);
char *line=NULL; size_t cap=0; ssize_t nr; char *line=NULL; size_t cap=0; ssize_t nr;
int pass=0, tot=0, dpass=0; int pass=0, tot=0, dpass=0;
while((nr=getline(&line,&cap,stdin))>=0){ while((nr=getline(&line,&cap,stdin))>=0){
+1 -1
View File
@@ -97,7 +97,7 @@ static void tok_load(Tok *T, const char *path){
jval *vocab=json_get(model,"vocab"); jval *vocab=json_get(model,"vocab");
jval *merges=json_get(model,"merges"); jval *merges=json_get(model,"merges");
jval *added=json_get(root,"added_tokens"); jval *added=json_get(root,"added_tokens");
if(!vocab||!merges){ fprintf(stderr,"tokenizer.json: manca model.vocab/merges\n"); exit(1); } if(!vocab||!merges){ fprintf(stderr,"tokenizer.json: missing model.vocab/merges\n"); exit(1); }
/* id massimo per dimensionare id2str */ /* id massimo per dimensionare id2str */
int maxid=0; int maxid=0;
+1 -1
View File
@@ -1,4 +1,4 @@
/* GENERATO da tools/gen_unicode.py — non modificare a mano. */ /* GENERATED by tools/gen_unicode.py — do not edit by hand. */
#ifndef TOK_UNICODE_H #ifndef TOK_UNICODE_H
#define TOK_UNICODE_H #define TOK_UNICODE_H
#include <stdint.h> #include <stdint.h>
+2 -2
View File
@@ -11,8 +11,8 @@ from pathlib import Path
SPEED_RE = re.compile(r"REPLAY decode:.*\| ([0-9.]+) tok/s") SPEED_RE = re.compile(r"REPLAY decode:.*\| ([0-9.]+) tok/s")
PROFILE_RE = re.compile( PROFILE_RE = re.compile(
r"PROFILO: expert-disk ([0-9.]+)s \| expert-matmul ([0-9.]+)s " r"PROFILE: expert-disk ([0-9.]+)s \| expert-matmul ([0-9.]+)s "
r"\| attention ([0-9.]+)s .* lm_head ([0-9.]+)s \| altro ([0-9.-]+)s" r"\| attention ([0-9.]+)s .* lm_head ([0-9.]+)s \| other ([0-9.-]+)s"
) )
PROFILE_KEYS = ("disk", "expert_matmul", "attention", "lm_head", "other") PROFILE_KEYS = ("disk", "expert_matmul", "attention", "lm_head", "other")
+32 -31
View File
@@ -139,11 +139,12 @@ def main():
ap.add_argument("--min-free-gb", type=float, default=20.0) ap.add_argument("--min-free-gb", type=float, default=20.0)
ap.add_argument("--selftest", action="store_true") ap.add_argument("--selftest", action="store_true")
ap.add_argument("--mtp", action="store_true", ap.add_argument("--mtp", action="store_true",
help="scarica/converte SOLO la testa MTP (model.layers.<n_layers>.*) -> out-mtp-*.safetensors") help="download and convert ONLY the MTP head (model.layers.<n_layers>.*) -> out-mtp-*.safetensors")
ap.add_argument("--indexer", action="store_true", ap.add_argument("--indexer", action="store_true",
help="estrae SOLO i pesi del DSA lightning indexer -> out-idx-*.safetensors. ATTENZIONE: " help="extract ONLY the DSA lightning-indexer weights -> out-idx-*.safetensors. WARNING: "
"i tensori indexer sono sparsi su ~tutti gli shard: ri-scarica l'intero repo (~756 GB " "indexer tensors are spread across nearly every shard, so this re-downloads the whole "
"di traffico) per tenerne pochi GB. Resumabile shard per shard. Consigliato --ebits 8.") "repository (~756 GB of traffic) to retain only a few GB. Resumable per shard. "
"Recommended: --ebits 8.")
a = ap.parse_args() a = ap.parse_args()
if a.ebits is None: if a.ebits is None:
# testa MTP a int4 = acceptance ~0-4% (misurato, issue #8): il draft sbaglia sempre # testa MTP a int4 = acceptance ~0-4% (misurato, issue #8): il draft sbaglia sempre
@@ -163,8 +164,8 @@ def main():
q = (w / sc.repeat_interleave(bs,0).repeat_interleave(bs,1)).to(torch.float8_e4m3fn) q = (w / sc.repeat_interleave(bs,0).repeat_interleave(bs,1)).to(torch.float8_e4m3fn)
deq = (q.to(torch.float32) * sc.repeat_interleave(bs,0).repeat_interleave(bs,1)) deq = (q.to(torch.float32) * sc.repeat_interleave(bs,0).repeat_interleave(bs,1))
rel = (deq - w).abs().mean() / w.abs().mean() rel = (deq - w).abs().mean() / w.abs().mean()
print(f"[selftest fp8 block-dequant] errore relativo medio = {rel:.4f} " print(f"[selftest fp8 block-dequant] mean relative error = {rel:.4f} "
f"({'OK' if rel < 0.05 else 'ALTO'})") f"({'OK' if rel < 0.05 else 'HIGH'})")
return return
os.makedirs(a.outdir, exist_ok=True) os.makedirs(a.outdir, exist_ok=True)
@@ -178,7 +179,7 @@ def main():
for fn in ["config.json"]: for fn in ["config.json"]:
src = os.path.join(a.indir, fn) src = os.path.join(a.indir, fn)
if os.path.exists(src): shutil.copy(src, a.outdir) if os.path.exists(src): shutil.copy(src, a.outdir)
print(f"convertito {len(shards)} shard -> {a.outdir}") print(f"converted {len(shards)} shards -> {a.outdir}")
return return
# reale: scarica shard per shard, converte, cancella # reale: scarica shard per shard, converte, cancella
@@ -211,7 +212,7 @@ def main():
lock = open(os.path.join(a.outdir, ".convert.lock"), "w") lock = open(os.path.join(a.outdir, ".convert.lock"), "w")
try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) try: fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError: except OSError:
print("ERRORE: un altro convertitore sta gia' lavorando su questa outdir. Esco."); return print("ERROR: another converter is already using this output directory. Exiting."); return
# dimensioni note dei file, riempite dopo repo_info: il downloader multi-stream le usa # dimensioni note dei file, riempite dopo repo_info: il downloader multi-stream le usa
# per calcolare i confini dei segmenti e per sapere quando un file e' completo. # per calcolare i confini dei segmenti e per sapere quando un file e' completo.
@@ -280,13 +281,13 @@ def main():
except Exception as ex: except Exception as ex:
with log_lock: with log_lock:
nres[0] += 1 nres[0] += 1
print(f" [dl] s{t}: {type(ex).__name__} a/at {(s0+done[t])/1e9:.2f} GB: " print(f" [dl] s{t}: {type(ex).__name__} at {(s0+done[t])/1e9:.2f} GB: "
f"riprendo/resuming (#{nres[0]})", flush=True) f"resuming (#{nres[0]})", flush=True)
_t.sleep(min(15, 1 + nres[0] // NS)) _t.sleep(min(15, 1 + nres[0] // NS))
th = [threading.Thread(target=worker, args=(t,), daemon=True) for t in range(NS)] th = [threading.Thread(target=worker, args=(t,), daemon=True) for t in range(NS)]
for x in th: x.start() for x in th: x.start()
print(f" [dl {_t.strftime('%H:%M:%S')}] connesso/connected: {NS} stream, " print(f" [dl {_t.strftime('%H:%M:%S')}] connected: {NS} streams, "
f"{sum(done)/1e9:.2f} di/of {expected/1e9:.2f} GB", flush=True) f"{sum(done)/1e9:.2f} of {expected/1e9:.2f} GB", flush=True)
mark = sum(done); tmark = t0 mark = sum(done); tmark = t0
while any(x.is_alive() for x in th): while any(x.is_alive() for x in th):
_t.sleep(5) _t.sleep(5)
@@ -309,7 +310,7 @@ def main():
os.replace(part, out) os.replace(part, out)
dt = max(_t.time() - t0, 1e-9) dt = max(_t.time() - t0, 1e-9)
print(f" [dl] {fn}: {expected/1e9:.2f} GB in {dt/60:.1f} min " print(f" [dl] {fn}: {expected/1e9:.2f} GB in {dt/60:.1f} min "
f"({expected/dt/1e6:.1f} MB/s medi/avg, {NS} stream, {nres[0]} riprese/resumes)", flush=True) f"({expected/dt/1e6:.1f} MB/s avg, {NS} streams, {nres[0]} resumes)", flush=True)
return out return out
def _download_single(url, fn, out, part, expected): def _download_single(url, fn, out, part, expected):
@@ -335,9 +336,9 @@ def main():
cl = r.headers.get("Content-Length") cl = r.headers.get("Content-Length")
if cl: expected = have + int(cl) if cl: expected = have + int(cl)
if have == 0 or nres: # segnale di vita subito / immediate sign of life if have == 0 or nres: # segnale di vita subito / immediate sign of life
print(f" [dl {_t.strftime('%H:%M:%S')}] connesso/connected" print(f" [dl {_t.strftime('%H:%M:%S')}] connected"
f"{f' @ {have/1e9:.2f} GB' if have else ''}" f"{f' @ {have/1e9:.2f} GB' if have else ''}"
f"{f' di/of {expected/1e9:.2f} GB' if expected else ''}", flush=True) f"{f' of {expected/1e9:.2f} GB' if expected else ''}", flush=True)
with open(part, "ab" if have else "wb") as f: with open(part, "ab" if have else "wb") as f:
if not have: f.truncate(0) if not have: f.truncate(0)
while True: while True:
@@ -357,16 +358,16 @@ def main():
except urllib.error.HTTPError as ex: except urllib.error.HTTPError as ex:
if ex.code == 416: break # gia' completo / already complete if ex.code == 416: break # gia' completo / already complete
nres += 1 nres += 1
print(f" [dl] HTTP {ex.code} a/at {have/1e9:.2f} GB: riprendo/resuming (#{nres})", flush=True) print(f" [dl] HTTP {ex.code} at {have/1e9:.2f} GB: resuming (#{nres})", flush=True)
_t.sleep(min(15, 1 + nres)) _t.sleep(min(15, 1 + nres))
except Exception as ex: except Exception as ex:
nres += 1 nres += 1
print(f" [dl] {type(ex).__name__} a/at {have/1e9:.2f} GB: riprendo/resuming (#{nres})", flush=True) print(f" [dl] {type(ex).__name__} at {have/1e9:.2f} GB: resuming (#{nres})", flush=True)
_t.sleep(min(15, 1 + nres)) _t.sleep(min(15, 1 + nres))
os.replace(part, out) os.replace(part, out)
dt = max(_t.time() - t0, 1e-9); sz = os.path.getsize(out) dt = max(_t.time() - t0, 1e-9); sz = os.path.getsize(out)
print(f" [dl] {fn}: {sz/1e9:.2f} GB in {dt/60:.1f} min " print(f" [dl] {fn}: {sz/1e9:.2f} GB in {dt/60:.1f} min "
f"({sz/dt/1e6:.1f} MB/s medi/avg, {nres} riprese/resumes)", flush=True) f"({sz/dt/1e6:.1f} MB/s avg, {nres} resumes)", flush=True)
return out return out
from safetensors.numpy import save_file from safetensors.numpy import save_file
@@ -380,7 +381,7 @@ def main():
break break
except KeyboardInterrupt: raise except KeyboardInterrupt: raise
except Exception as ex: except Exception as ex:
w = min(60, 5*(att+1)); print(f"repo_info KO ({type(ex).__name__}): riprovo tra {w}s", flush=True); _t.sleep(w) w = min(60, 5*(att+1)); print(f"repo_info failed ({type(ex).__name__}); retrying in {w}s", flush=True); _t.sleep(w)
shards = sorted(s.rfilename for s in info.siblings if s.rfilename.endswith(".safetensors")) shards = sorted(s.rfilename for s in info.siblings if s.rfilename.endswith(".safetensors"))
for fn in ["config.json", "tokenizer.json", "tokenizer_config.json", "generation_config.json"]: for fn in ["config.json", "tokenizer.json", "tokenizer_config.json", "generation_config.json"]:
try: shutil.copy(hf_hub_download(a.repo, fn, local_dir=a.outdir+"/_meta"), a.outdir) try: shutil.copy(hf_hub_download(a.repo, fn, local_dir=a.outdir+"/_meta"), a.outdir)
@@ -392,19 +393,19 @@ def main():
f"https://huggingface.co/{a.repo}/resolve/main/model.safetensors.index.json", timeout=30).read())["weight_map"] f"https://huggingface.co/{a.repo}/resolve/main/model.safetensors.index.json", timeout=30).read())["weight_map"]
pref = f"model.layers.{a.n_layers}." pref = f"model.layers.{a.n_layers}."
mtp_shards = sorted(set(v for k, v in idx.items() if k.startswith(pref))) mtp_shards = sorted(set(v for k, v in idx.items() if k.startswith(pref)))
print(f"[MTP] testa nel layer {a.n_layers}: {len(mtp_shards)} shard da processare: {mtp_shards}") print(f"[MTP] head at layer {a.n_layers}: {len(mtp_shards)} shards to process: {mtp_shards}")
for i, sh in enumerate(mtp_shards): for i, sh in enumerate(mtp_shards):
outp = os.path.join(a.outdir, f"out-mtp-{i:05d}.safetensors") outp = os.path.join(a.outdir, f"out-mtp-{i:05d}.safetensors")
if os.path.exists(outp): print(f"[MTP] {outp} gia' fatto"); continue if os.path.exists(outp): print(f"[MTP] {outp} already done"); continue
print(f"[MTP {i+1}/{len(mtp_shards)}] scarico {sh}...", flush=True) print(f"[MTP {i+1}/{len(mtp_shards)}] downloading {sh}...", flush=True)
p = download_retry(a.repo, sh, tmp) p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_mtp=True) out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_mtp=True)
save_file(out, outp) save_file(out, outp)
os.remove(p) os.remove(p)
for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True): for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True):
if os.path.isfile(blob): os.remove(blob) if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB, {len(out)} tensori)", flush=True) print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB, {len(out)} tensors)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[MTP] FATTO."); return shutil.rmtree(tmp, ignore_errors=True); print("[MTP] DONE."); return
if a.indexer: if a.indexer:
import urllib.request import urllib.request
idx = json.loads(urllib.request.urlopen( idx = json.loads(urllib.request.urlopen(
@@ -412,25 +413,25 @@ def main():
idx_shards = sorted(set(v for k, v in idx.items() idx_shards = sorted(set(v for k, v in idx.items()
if "indexer" in k and 0 <= layer_idx(k) < a.n_layers)) if "indexer" in k and 0 <= layer_idx(k) < a.n_layers))
tot_gb = len(idx_shards) * 5.4 tot_gb = len(idx_shards) * 5.4
print(f"[IDX] pesi indexer su {len(idx_shards)} shard (~{tot_gb:.0f} GB di download totale, resumabile)") print(f"[IDX] indexer weights across {len(idx_shards)} shards (~{tot_gb:.0f} GB total download, resumable)")
for i, sh in enumerate(idx_shards): for i, sh in enumerate(idx_shards):
outp = os.path.join(a.outdir, f"out-idx-{i:05d}.safetensors") outp = os.path.join(a.outdir, f"out-idx-{i:05d}.safetensors")
if os.path.exists(outp): continue # gia' fatto -> ripartibile if os.path.exists(outp): continue # gia' fatto -> ripartibile
print(f"[IDX {i+1}/{len(idx_shards)}] scarico {sh}...", flush=True) print(f"[IDX {i+1}/{len(idx_shards)}] downloading {sh}...", flush=True)
p = download_retry(a.repo, sh, tmp) p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_idx=True) out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits, keep_idx=True)
if out: save_file(out, outp) if out: save_file(out, outp)
os.remove(p) os.remove(p)
for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True): for blob in glob.glob(os.path.join(tmp, "**", "*"), recursive=True):
if os.path.isfile(blob): os.remove(blob) if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({len(out)} tensori)", flush=True) print(f" -> {os.path.basename(outp)} ({len(out)} tensors)", flush=True)
shutil.rmtree(tmp, ignore_errors=True); print("[IDX] FATTO."); return shutil.rmtree(tmp, ignore_errors=True); print("[IDX] DONE."); return
for i, sh in enumerate(shards): for i, sh in enumerate(shards):
if free_gb(a.outdir) < a.min_free_gb: if free_gb(a.outdir) < a.min_free_gb:
print(f"STOP: spazio libero < {a.min_free_gb} GB. Libera spazio e rilancia (riprende)."); break print(f"STOP: free space is below {a.min_free_gb} GB. Free space and rerun to resume."); break
outp = os.path.join(a.outdir, f"out-{i:05d}.safetensors") outp = os.path.join(a.outdir, f"out-{i:05d}.safetensors")
if os.path.exists(outp): continue # gia' fatto -> ripartibile if os.path.exists(outp): continue # gia' fatto -> ripartibile
print(f"[{i+1}/{len(shards)}] scarico {sh} (libero {free_gb(a.outdir):.0f} GB)...", flush=True) print(f"[{i+1}/{len(shards)}] downloading {sh} ({free_gb(a.outdir):.0f} GB free)...", flush=True)
p = download_retry(a.repo, sh, tmp) p = download_retry(a.repo, sh, tmp)
out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits) out = {}; convert_shard(p, out, a.n_layers, a.ebits, a.io_bits, a.xbits)
save_file(out, outp) save_file(out, outp)
@@ -439,7 +440,7 @@ def main():
if os.path.isfile(blob): os.remove(blob) if os.path.isfile(blob): os.remove(blob)
print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB)", flush=True) print(f" -> {os.path.basename(outp)} ({os.path.getsize(outp)/1e9:.2f} GB)", flush=True)
shutil.rmtree(tmp, ignore_errors=True) shutil.rmtree(tmp, ignore_errors=True)
print("FATTO." if i == len(shards)-1 else "INTERROTTO (rilancia per riprendere).") print("DONE." if i == len(shards)-1 else "INTERRUPTED (rerun to resume).")
if __name__ == "__main__": if __name__ == "__main__":
main() main()
+6 -6
View File
@@ -28,15 +28,15 @@ def check():
sts = [s for s in info.siblings if s.rfilename.endswith(".safetensors")] sts = [s for s in info.siblings if s.rfilename.endswith(".safetensors")]
free = shutil.disk_usage(os.path.dirname(DEST) or "/").free free = shutil.disk_usage(os.path.dirname(DEST) or "/").free
print(f"repo: {REPO}") print(f"repo: {REPO}")
print(f" file totali: {len(info.siblings)} ({len(sts)} shard safetensors)") print(f" total files: {len(info.siblings)} ({len(sts)} safetensors shards)")
print(f" dimensione totale: {human(tot)}") print(f" total size: {human(tot)}")
print(f" spazio libero in {DEST}: {human(free)}") print(f" free space in {DEST}: {human(free)}")
print(f" {'OK: ci sta' if free > tot*1.05 else 'ATTENZIONE: spazio insufficiente'}") print(f" {'OK: enough space' if free > tot*1.05 else 'WARNING: not enough space'}")
def download(): def download():
os.makedirs(DEST, exist_ok=True) os.makedirs(DEST, exist_ok=True)
free = shutil.disk_usage(DEST).free free = shutil.disk_usage(DEST).free
print(f"Scarico {REPO} -> {DEST} (libero: {human(free)})") print(f"Downloading {REPO} -> {DEST} (free: {human(free)})")
# resume_download e' implicito; in caso di interruzione, rilancia e riprende. # resume_download e' implicito; in caso di interruzione, rilancia e riprende.
snapshot_download( snapshot_download(
repo_id=REPO, repo_id=REPO,
@@ -44,7 +44,7 @@ def download():
allow_patterns=["*.safetensors", "*.json", "*.txt", "*.model"], allow_patterns=["*.safetensors", "*.json", "*.txt", "*.model"],
max_workers=8, max_workers=8,
) )
print("FATTO. Pesi in:", DEST) print("DONE. Weights saved in:", DEST)
if __name__ == "__main__": if __name__ == "__main__":
if "--check" in sys.argv: if "--check" in sys.argv:
+15 -15
View File
@@ -43,7 +43,7 @@ def load_docs(task, data_dir, limit, seed):
return SMOKE[:limit] if limit else SMOKE return SMOKE[:limit] if limit else SMOKE
path = os.path.join(data_dir, task + ".jsonl") path = os.path.join(data_dir, task + ".jsonl")
if not os.path.exists(path): if not os.path.exists(path):
sys.exit(f"manca {path} — generalo con: python3 tools/fetch_benchmarks.py --out {data_dir} --tasks {task}") sys.exit(f"missing {path} — generate it with: python3 tools/fetch_benchmarks.py --out {data_dir} --tasks {task}")
docs = [json.loads(l) for l in open(path) if l.strip()] docs = [json.loads(l) for l in open(path) if l.strip()]
random.Random(seed).shuffle(docs) random.Random(seed).shuffle(docs)
return docs[:limit] if limit else docs return docs[:limit] if limit else docs
@@ -84,9 +84,9 @@ def score_accuracy(tasks, meta, perq, lp):
print(f"{t:<18} {n:>4} {100*acc/n:>6.1f}% {100*accn/n:>8.1f}%") print(f"{t:<18} {n:>4} {100*acc/n:>6.1f}% {100*accn/n:>8.1f}%")
overall.append(100 * accn / n) overall.append(100 * accn / n)
for mdl, sc in REFERENCE.get(t, {}).items(): for mdl, sc in REFERENCE.get(t, {}).items():
if sc is not None: print(f"{' rif '+mdl:<18} {'':>4} {'':>7} {sc:>8.1f}%") if sc is not None: print(f"{' ref '+mdl:<18} {'':>4} {'':>7} {sc:>8.1f}%")
if overall: if overall:
print(f"\nMEDIA acc_norm: {sum(overall)/len(overall):.1f}% su {len(overall)} task") print(f"\nMEAN acc_norm: {sum(overall)/len(overall):.1f}% across {len(overall)} tasks")
def main(): def main():
ap = argparse.ArgumentParser() ap = argparse.ArgumentParser()
@@ -99,8 +99,8 @@ def main():
ap.add_argument("--cap", type=int, default=64) ap.add_argument("--cap", type=int, default=64)
ap.add_argument("--bits", default="") ap.add_argument("--bits", default="")
ap.add_argument("--seed", type=int, default=1234) ap.add_argument("--seed", type=int, default=1234)
ap.add_argument("--dry", action="store_true", help="costruisci le richieste e fermati (no motore)") ap.add_argument("--dry", action="store_true", help="build requests and stop without running the engine")
ap.add_argument("--selftest", action="store_true", help="verifica la matematica dello scoring") ap.add_argument("--selftest", action="store_true", help="verify the scoring calculations")
a = ap.parse_args() a = ap.parse_args()
if a.selftest: # acc/acc_norm con logprob sintetici if a.selftest: # acc/acc_norm con logprob sintetici
@@ -113,32 +113,32 @@ def main():
tk = Tokenizer.from_file(os.path.join(a.snap, "tokenizer.json")) tk = Tokenizer.from_file(os.path.join(a.snap, "tokenizer.json"))
tasks = [t.strip() for t in a.tasks.split(",") if t.strip()] tasks = [t.strip() for t in a.tasks.split(",") if t.strip()]
docs_by_task = {t: load_docs(t, a.data, a.limit, a.seed) for t in tasks} docs_by_task = {t: load_docs(t, a.data, a.limit, a.seed) for t in tasks}
for t, d in docs_by_task.items(): print(f"[{t}] {len(d)} domande", file=sys.stderr) for t, d in docs_by_task.items(): print(f"[{t}] {len(d)} questions", file=sys.stderr)
reqs, meta, perq = build_requests(tk, docs_by_task) reqs, meta, perq = build_requests(tk, docs_by_task)
print(f"richieste totali: {len(reqs)} (opzioni)", file=sys.stderr) print(f"total requests: {len(reqs)} (answer options)", file=sys.stderr)
if a.dry: if a.dry:
for r in reqs[:3]: print(" esempio req:", r[:80], "...", file=sys.stderr) for r in reqs[:3]: print(" example request:", r[:80], "...", file=sys.stderr)
print("DRY: meccanica ok (tokenizzazione+richieste). Niente motore.", file=sys.stderr); return print("DRY: request construction and tokenization passed. Engine was not run.", file=sys.stderr); return
req_path = tempfile.mktemp(suffix=".txt") req_path = tempfile.mktemp(suffix=".txt")
open(req_path, "w").write("\n".join(reqs) + "\n") open(req_path, "w").write("\n".join(reqs) + "\n")
env = dict(os.environ, SNAP=a.snap, SCORE=req_path) env = dict(os.environ, SNAP=a.snap, SCORE=req_path)
if a.ram: env["RAM_GB"] = str(a.ram) if a.ram: env["RAM_GB"] = str(a.ram)
cmd = [a.glm, str(a.cap)] + a.bits.split() cmd = [a.glm, str(a.cap)] + a.bits.split()
print("eseguo:", " ".join(cmd), file=sys.stderr) print("running:", " ".join(cmd), file=sys.stderr)
t0 = time.time() t0 = time.time()
proc = subprocess.run(cmd, env=env, capture_output=True, text=True) proc = subprocess.run(cmd, env=env, capture_output=True, text=True)
if proc.returncode != 0: if proc.returncode != 0:
print("ERRORE motore:\n", proc.stderr[-2000:], file=sys.stderr); sys.exit(1) print("ENGINE ERROR:\n", proc.stderr[-2000:], file=sys.stderr); sys.exit(1)
lines = [l for l in proc.stdout.strip().splitlines() if l and l[0] in "-0123456789"] lines = [l for l in proc.stdout.strip().splitlines() if l and l[0] in "-0123456789"]
if len(lines) != len(reqs): if len(lines) != len(reqs):
print(f"ATTENZIONE: {len(lines)} output vs {len(reqs)} richieste", file=sys.stderr) print(f"WARNING: {len(lines)} outputs for {len(reqs)} requests", file=sys.stderr)
lp = [float(l.split()[0]) for l in lines] lp = [float(l.split()[0]) for l in lines]
print(f"(motore: {time.time()-t0:.0f}s){proc.stderr.strip().splitlines()[-1] if proc.stderr.strip() else ''}", file=sys.stderr) print(f"(engine: {time.time()-t0:.0f}s){proc.stderr.strip().splitlines()[-1] if proc.stderr.strip() else ''}", file=sys.stderr)
score_accuracy(tasks, meta, perq, lp) score_accuracy(tasks, meta, perq, lp)
print("\nNB: confronta acc_norm col punteggio PUBBLICATO di GLM-5.2 (model card). Se vicino," print("\nNOTE: compare acc_norm with GLM-5.2's PUBLISHED model-card score. A close result"
"\n la quantizzazione int4 ha preservato il modello. (riempi REFERENCE in tools/eval_glm.py)") "\n indicates that int4 quantization preserved quality. (Fill REFERENCE in tools/eval_glm.py.)")
os.remove(req_path) os.remove(req_path)
if __name__ == "__main__": if __name__ == "__main__":
+1 -1
View File
@@ -48,7 +48,7 @@ def main():
from datasets import load_dataset from datasets import load_dataset
os.makedirs(a.out, exist_ok=True) os.makedirs(a.out, exist_ok=True)
for t in [x.strip() for x in a.tasks.split(",") if x.strip()]: for t in [x.strip() for x in a.tasks.split(",") if x.strip()]:
if t not in TASKS: print("task ignoto:", t); continue if t not in TASKS: print("unknown task:", t); continue
path, cfg, split, fn = TASKS[t] path, cfg, split, fn = TASKS[t]
ds = load_dataset(path, cfg, split=split) ds = load_dataset(path, cfg, split=split)
idx = list(range(len(ds))); random.Random(a.seed).shuffle(idx) idx = list(range(len(ds))); random.Random(a.seed).shuffle(idx)
+1 -1
View File
@@ -41,7 +41,7 @@ def emit(name, rs):
print("};") print("};")
print(f"static const int {name}_n = {len(rs)};\n") print(f"static const int {name}_n = {len(rs)};\n")
print("/* GENERATO da tools/gen_unicode.py — non modificare a mano. */") print("/* GENERATED by tools/gen_unicode.py — do not edit by hand. */")
print("#ifndef TOK_UNICODE_H\n#define TOK_UNICODE_H\n#include <stdint.h>\n") print("#ifndef TOK_UNICODE_H\n#define TOK_UNICODE_H\n#include <stdint.h>\n")
emit("uni_L", L); emit("uni_N", N); emit("uni_S", S) emit("uni_L", L); emit("uni_N", N); emit("uni_S", S)
print("""static int uni_in(const uint32_t t[][2], int n, uint32_t cp){ print("""static int uni_in(const uint32_t t[][2], int n, uint32_t cp){
+2 -2
View File
@@ -53,7 +53,7 @@ with torch.no_grad():
layer.mlp.gate.e_score_correction_bias.copy_( layer.mlp.gate.e_score_correction_bias.copy_(
torch.linspace(-0.1, 0.1, cfg.n_routed_experts)) torch.linspace(-0.1, 0.1, cfg.n_routed_experts))
print("=== tensori dello state_dict (nomi per il loader C) ===") print("=== state_dict tensors (names used by the C loader) ===")
for n, p in model.state_dict().items(): for n, p in model.state_dict().items():
print(f" {n:60s} {tuple(p.shape)}") print(f" {n:60s} {tuple(p.shape)}")
@@ -76,4 +76,4 @@ print("tf_pred:", tf_pred)
model.save_pretrained("glm_tiny", safe_serialization=True) model.save_pretrained("glm_tiny", safe_serialization=True)
json.dump(cfg.to_dict(), open("glm_tiny/config.json", "w")) json.dump(cfg.to_dict(), open("glm_tiny/config.json", "w"))
json.dump({"prompt_ids": prompt, "full_ids": full, "tf_pred": tf_pred}, open("ref_glm.json", "w")) json.dump({"prompt_ids": prompt, "full_ids": full, "tf_pred": tf_pred}, open("ref_glm.json", "w"))
print("\nsalvato: glm_tiny/ (pesi+config) e ref_glm.json") print("\nsaved: glm_tiny/ (weights + config) and ref_glm.json")