**Tiny engine, immense model.** Run **GLM-5.2 (744B-parameter MoE)** on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk. ``` $ ./coli chat 🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ pronto in 32s · residente 9.9 GB › ciao! ◆ Ciao! 😊 Come posso aiutarti oggi? ``` ## The idea A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So: - the **dense part** (attention, shared experts, embeddings — ~17B params) stays **resident in RAM at int4** (~9.9 GB); - the **21,504 routed experts** (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live **on disk** (~370 GB) and are **streamed on demand**, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2. The engine is a single C file (`c/glm.c`, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU. ## What's implemented - **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture). - **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA). - **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers. - **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. Measured **2.00 tokens/forward (100% acceptance)** on structured text. Lossless: output identical to greedy. - **Quantization kernels**: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container. - **Batch-union MoE**: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it. - **Byte-level BPE tokenizer in C** (GPT-2-style with Unicode-property regex, 320k merges). - **RAM safety**: the expert cache is auto-sized from `MemAvailable` at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires. - **Offline FP8→int4 converter** (`c/convert_fp8_to_int4.py`): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable. ## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX) | metric | value | |---|---| | model on disk (int4 container) | ~370 GB | | resident RAM (dense, int4) | 9.9 GB | | load time | ~30 s | | peak RSS during chat | ~20 GB (auto-capped) | | cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) | | disk ceiling (VHDX random) | ~1 GB/s → ~0.05–0.1 tok/s cold | | MTP speculation | 2.0 tok/forward measured | This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest. ### SSD Wear Warning Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health. ### Quick start ```bash cd c ./setup.sh # checks gcc/OpenMP, builds, self-tests # ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard # (never needs the full 756 GB at once), converts to the int4 container, then # converts the MTP head for speculative decoding. Resumable at any point. # Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy ./coli convert --model /nvme/glm52_i4 # ~400 GB free on a real ext4/NVMe path # chat — RAM budget, expert cache and MTP are all detected automatically: COLI_MODEL=/nvme/glm52_i4 ./coli chat ``` The engine at runtime is pure C — python is only used by the one-time converter. Useful knobs (env or flags): `--temp 0.7` token sampling temperature (0 = greedy, default 1.0 + nucleus 0.95), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:piu` in chat continues a truncated one), `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `TF=1` teacher-forcing validation. **The learning cache**: the engine records which experts your usage actually routes to (`.coli_usage` next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it. ## Got a better machine? Try it — here's what to expect colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB/s). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4 — never a network/9p mount). **How to test it, in order:** ```bash cd c && ./setup.sh # build + architecture self-test (expects 32/32) # 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads): gcc -O2 -fopenmp iobench.c -o iobench ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads ./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT # 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS): COLI_MODEL=/path/to/glm52_i4 ./coli chat # 3) record expert usage, then pin the hottest experts in your spare RAM: STATS=stats.txt ./coli chat PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM # 4) quality benchmarks (MMLU/HellaSwag/ARC): ./coli bench ``` **Back-of-envelope predictions** (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost; RAM turns cold reads into free cache hits): | machine | expected | |---|---| | this dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline | | native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s | | PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s | | 128–256 GB RAM workstation (hot expert set mostly cached) | ~5–15 tok/s, matmul-bound — interactive | These are estimates, not measurements — if you run colibrì on serious hardware, **please open an issue with your numbers**: real datapoints from better machines are exactly what this project needs next. ## Supporting the project colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates *directly* into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can: - ⭐ star the repo and share it; - 🐛 open issues with benchmark numbers from your hardware; - 💬 reach out via GitHub issues if you'd like to sponsor development or donate hardware. Every contribution, from a datapoint to a disk, moves the ceiling. ## Repo layout ``` c/glm.c the engine (GLM-5.2 forward, streaming MoE, MTP, serve mode) c/st.h safetensors reader: pread + fadvise, no mmap (RSS stays flat) c/tok.h byte-level BPE tokenizer in C c/coli CLI: chat / run / bench / convert / info c/iobench.c parallel disk microbenchmark (measures what the engine feels) c/convert_fp8_to_int4.py disk-safe FP8 → int4 converter c/make_glm_oracle.py tiny-random oracle generator for validation c/olmoe.c stage-A engine (OLMoE), first validation target ``` ## Why "colibrì" The hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. This engine keeps a 744-billion-parameter giant alive on hummingbird rations: 25 GB of RAM, twelve CPU cores, and a lot of disk patience. ## License Apache 2.0. GLM-5.2 weights are released by Z.ai under MIT.