diff --git a/README.md b/README.md index be1e709..63e7a3f 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ The engine is a single C file (`c/glm.c`, ~1,300 lines) plus small headers. No B - **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture). - **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA). - **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers. -- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. Measured **2.00 tokens/forward (100% acceptance)** on structured text. Lossless — *and stays lossless under sampling* via rejection sampling (accept draft w.p. p(draft); on reject, resample with the draft banned). +- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless — *and stays lossless under sampling* via rejection sampling. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up — the adaptive guard and `DRAFT=0` are there for that. - **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail). - **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower). - **MLA weight absorption** (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs `kv_b`, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere. @@ -48,7 +48,7 @@ The engine is a single C file (`c/glm.c`, ~1,300 lines) plus small headers. No B | peak RSS during chat | ~20 GB (auto-capped) | | cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) | | disk ceiling (VHDX random) | ~1 GB/s → ~0.05–0.1 tok/s cold | -| MTP speculation | 2.0 tok/forward measured | +| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) | This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest. @@ -118,7 +118,7 @@ PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM ./coli bench ``` -**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost; RAM turns cold reads into free cache hits): +**Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost *once the cache is warm*; RAM turns cold reads into free cache hits): | machine | expected | |---|---| @@ -138,8 +138,9 @@ Real numbers from real machines, stock build (`setup.sh`, gcc 13), greedy decodi |---|---|---|---| | Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX ([#2](https://github.com/JustVugg/colibri/issues/2)) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB | | 〃 | 〃 | `--topp 0.7` | **0.11 tok/s** · expert hit 11% · RSS 14.7 GB | +| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD ([#4](https://github.com/JustVugg/colibri/issues/4), [#5](https://github.com/JustVugg/colibri/issues/5)) | 14.2 GB/s O_DIRECT | default, MTP off | **1.06 tok/s** · expert hit 23% · RSS 21.8 GB | -Takeaways from this datapoint: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7× faster than the dev box — **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts. And `--topp 0.7` alone bought a clean 1.6× end-to-end speedup. +Takeaways: with 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so decode stays cold even on a disk 2–2.7× faster than the dev box — **on small-RAM machines the RAM cap, not the disk, is the binding constraint**, exactly as the table above predicts; `--topp 0.7` alone bought a clean 1.6× end-to-end speedup. The M5 Max datapoint lands right on the table's second row: **~1 tok/s of a 744B model on a laptop SSD** — and its 14 GB/s disk shifts the bottleneck back to RAM budget and kernels. ## Quality benchmark — help wanted diff --git a/c/coli b/c/coli index 36b75fc..2cfdf40 100644 --- a/c/coli +++ b/c/coli @@ -409,9 +409,11 @@ def cmd_convert(a): rc=subprocess.call(base) if rc!=0: sys.exit(rc) if a.no_mtp: sys.exit(0) - # passo 2: testa MTP (layer 78) -> abilita la decodifica speculativa nativa (piu' veloce) - print(f" {C.dim}[2/2] testa MTP (draft speculativi){C.r}") - sys.exit(subprocess.call(base+["--mtp"])) + # passo 2: testa MTP (layer 78). SEMPRE int8: a int4 i draft sbagliano quasi sempre + # (acceptance 0-4% vs 39-59%, misurato — issue #8) e la speculazione non parte mai. + mtp_cmd=list(base); i=mtp_cmd.index("--ebits"); mtp_cmd[i+1]=str(max(8,a.ebits)) + print(f" {C.dim}[2/2] testa MTP a int8 (draft speculativi){C.r}") + sys.exit(subprocess.call(mtp_cmd+["--mtp"])) def main(): common=argparse.ArgumentParser(add_help=False) diff --git a/c/convert_fp8_to_int4.py b/c/convert_fp8_to_int4.py index f77e91e..5785ee9 100644 --- a/c/convert_fp8_to_int4.py +++ b/c/convert_fp8_to_int4.py @@ -132,7 +132,7 @@ def main(): ap.add_argument("--repo", default=None) ap.add_argument("--indir", default=None) ap.add_argument("--outdir", required=False) - ap.add_argument("--ebits", type=int, default=4) # bit residenti: attn/dense-mlp/shared + ap.add_argument("--ebits", type=int, default=None) # bit residenti (default 4; 8 per --mtp/--indexer) ap.add_argument("--io-bits", type=int, default=8) # bit di embed/lm_head ap.add_argument("--xbits", type=int, default=None) # bit degli expert ROUTED (streaming); default=ebits ap.add_argument("--n-layers", type=int, default=78) @@ -145,6 +145,10 @@ def main(): "i tensori indexer sono sparsi su ~tutti gli shard: ri-scarica l'intero repo (~756 GB " "di traffico) per tenerne pochi GB. Resumabile shard per shard. Consigliato --ebits 8.") a = ap.parse_args() + if a.ebits is None: + # testa MTP a int4 = acceptance ~0-4% (misurato, issue #8): il draft sbaglia sempre + # e la speculazione non parte mai. A int8: 39-59%, 2.2-2.8 token/forward. + a.ebits = 8 if (a.mtp or a.indexer) else 4 if a.xbits is None: a.xbits = a.ebits if a.selftest: