From 3fb5a001060c885259558fbd7cca4741e0ac49d2 Mon Sep 17 00:00:00 2001 From: ZacharyZcR Date: Mon, 13 Jul 2026 04:06:51 +0800 Subject: [PATCH] =?UTF-8?q?docs:=20GLM-5.2=206x=20RTX=205090=20full-reside?= =?UTF-8?q?ncy=20experiment=20=E2=80=94=206.84=20tok/s,=20honest=20dead-en?= =?UTF-8?q?ds=20log=20(#94)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the July 12-13 lab findings on one 6x RTX 5090 machine: - vLLM-Moet comparison and why per-rank expert residency dominates - colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the tested-and-rejected directions documented - MTP speculation: broken int4 head identified (issue #8), int8 head reaches 69-79% acceptance but MoE verify batches scale expert cost with S, so speculation loses at every depth; revisit after GPU grouped GEMM - AVX-512 int4 kernel qualification: numerically better than the old order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing README links the record from the honest-numbers section. --- README.md | 2 + docs/experiments/glm52-6x5090-2026-07-12.md | 302 ++++++++++++++++++++ 2 files changed, 304 insertions(+) create mode 100644 docs/experiments/glm52-6x5090-2026-07-12.md diff --git a/README.md b/README.md index b8d3dc6..66bf1ef 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B ## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX) +Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode. + | metric | value | |---|---| | model on disk (int4 container) | ~370 GB | diff --git a/docs/experiments/glm52-6x5090-2026-07-12.md b/docs/experiments/glm52-6x5090-2026-07-12.md new file mode 100644 index 0000000..2c86c7c --- /dev/null +++ b/docs/experiments/glm52-6x5090-2026-07-12.md @@ -0,0 +1,302 @@ +# GLM-5.2 on 6x RTX 5090 + +This note records the July 12, 2026 experiments that compared colibri with +vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon +Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe +storage. It exists to preserve the measured facts and avoid repeating the same +dead ends. + +The day had two halves. The morning runs (below) measured colibri at 0.12 +tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on +colibri (see "colibri full-resident experiments") removed disk from the decode +path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark +and 6.84 tok/s on a 256-token run — the fastest single-request result measured +on this machine so far. + +## Models and artifacts + +- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards) +- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards) +- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB) +- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB) +- vLLM-Moet checkout: `/data/test/vLLM-Moet` +- Built image: `vllm-moet-sm120:v024` + +The official model was downloaded through `hf-mirror.com` with +`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled; +ordinary HTTP from the mirror sustained roughly 22-46 MB/s. + +## Kernel validation + +All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke +tests. Representative correctness results were: + +| Path | Shape | Max relative error | Cosine similarity | +| --- | --- | ---: | ---: | +| Decode | T=2 | 2.086e-2 | 0.999878 | +| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 | +| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 | + +The kernels work on SM120. The performance limitation below is therefore not +an unsupported-GPU fallback. + +## Real generation results + +All decode rates below are for a single request, not aggregate throughput. + +| Runtime and layout | GPU use | TTFT | Decode rate | Important observation | +| --- | --- | ---: | ---: | --- | +| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU | +| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | — | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits | +| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay | +| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% | +| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse | + +The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only +about 0.1 tok/s because every decode step still missed experts and replayed the +whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not +reduce replay enough to offset pipeline and PYNCCL communication costs. + +The morning colibri run generated 16 tokens in 132.46 seconds. Its measured +hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the +expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk +reads (29.31 s), and attention (14.28 s). The afternoon experiments below +showed the dominant problem was expert loading from disk, not the CUDA path +itself. + +## colibri full-resident experiments (afternoon) + +Test directory: `/data/test/colibri-full-resident`. Related PR: +[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B +runs used the same binary flags, environment, prompt, `TEMP=0`, and token +count; only decode throughput is compared, never load or prefill time. + +### The optimization ladder (each step cumulative) + +| Change | Decode rate | Evidence | +| --- | ---: | --- | +| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens | +| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits | +| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s → 5.96 s | +| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 → pinned 5.08 tok/s on the same prompt | +| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 → 37,285 | +| Decode swap cap 32 → 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 → 0.09 s/round | + +The final configuration measured 6.28 tok/s on the fixed 96-token greedy +benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed +costs; always state the token count when quoting a rate). + +Winning layout, found automatically: + +```text +GPU experts: 9,343 / 19,456 (176.73 GB across six cards) +RAM experts: 10,113 / 19,456 (about 191.3 GB) +Disk service/wait during decode: 0 s +``` + +Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto +CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with +`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default +generated by `resource_plan.py`). + +### Directions tested and rejected (all reverted) + +- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per + round: swap/sync overhead or stale GPU expert sets lose more than they gain. + 16 tokens between repins and 16 swaps per round is the current local optimum + (8 swaps also regresses: fewer GPU calls outweigh the sync savings). +- A second prefill correction pass: overfits the short prompt's routing and + disturbs the history-based layout (5.86 tok/s). +- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery + (6.15 tok/s): both cost as much as re-reading from page cache. +- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30 + tok/s); GPU expert groups are too small a fraction of step time, and the + extra threads compete with the 24 CPU workers. +- OpenMP restructuring of RAM experts (row batching across experts, + persistent teams): neutral or negative; libgomp already reuses hot teams, + and interleaving experts in one static schedule destroys per-expert + NUMA/cache locality. +- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs, + smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI + int4×int8 decode: all neutral or negative. Capacity and TLB are not the + limit; raw history counts predict routing better than normalized ones. +- Next-layer expert prediction with GPU staging: the routing signal is real + (70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the + expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth + revisiting only with dedicated streams that do not stall attention. + +### Candidate not yet merged: AVX-512 int4 kernel + +The lossless int4→float RAM expert kernel still uses 256-bit AVX2. A 512-bit +unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 → 5.89 +tok/s, expert time −19.7%), and a dual-accumulator variant added a little more +(5.89 → 5.94 tok/s at 96 tokens). However, the changed accumulation order +produces small float differences and the 64-token outputs eventually diverge, +so it cannot be claimed bit-exact. It stays out of the tree until it has +numeric-error unit tests, a perplexity/token-consistency evaluation, and an +explicit decision on whether float-accumulation differences are acceptable. +Row blocking (2/4 output rows) looked good in short microbenchmarks but the +gain disappeared over 10,000 sustained iterations and in a 256-token run +(6.84 vs 6.83 tok/s) — a turbo/cache artifact, reverted. + +### Where the time goes now + +At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other +2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of +CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE +sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention +−5.0%) with no math change. Scheduling-level CPU optimizations are exhausted — +the remaining expert time is the int4 matmul kernel itself. + +## MTP speculative decoding at full residency (July 13) + +Motivation: with all experts resident and disk waits at zero, the old +objection to speculation (a rejected draft wastes a disk read) no longer +applies, so the July 3 negative result deserved a retest. All runs used the +same binary and environment as the 16-swap configuration above, `TEMP=0`, +96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy +output degenerates into repetition, and a non-degenerate "water cycle" +prompt). + +Phase 1 found the installed MTP head unusable: acceptance 0-4% at every +depth, exactly the known int4-head defect from issue #8 (the three +`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint). +Speculation with the broken head costs 15-18% throughput — pure verification +tax. + +Phase 2 replaced the head with the community int8 MTP shards +(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as +`.int4.bak`). The head then worked as designed — and speculation still lost +at every depth: + +| Prompt | DRAFT | tok/s | tok/forward | Acceptance | +| --- | ---: | ---: | ---: | ---: | +| water cycle | 0 | 6.79 | 1.01 | — | +| water cycle | 1 | 6.45 | 1.81 | 79% | +| water cycle | 2 | 5.65 | 2.29 | 64% | +| water cycle | 3 | 6.13 | 3.10 | 69% | +| sky (degenerate) | 0 | 6.12 | 1.01 | — | +| sky (degenerate) | 1 | 5.51 | 1.75 | 73% | +| sky (degenerate) | 2 | 4.42 | 1.88 | 44% | +| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% | + +The failure is structural, not a tuning problem. In a dense model a verify +batch of S tokens rereads the same weights once, so speculation amortizes +the dominant cost. In this MoE the dominant cost is per-unique-expert weight +processing, and the S positions of a verify batch route to mostly different +experts: measured per-forward expert time scaled almost linearly with S +(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt). +There is no amortization to collect, and the MTP head's own forward (int8 +experts, one extra layer per draft token) adds cost on top. Even 79% +acceptance at DRAFT=1 nets -5%. + +Two useful facts survive the negative result. First, the int8 head is +genuinely good: 69-79% chained top-1 acceptance on clean text (the int8 +shards stay installed; the int4 originals were defective). Second, the +conclusion inverts if routed-expert compute per extra position ever becomes +cheap — e.g. Tensor Core grouped GEMM computing an S=4 verify batch at +near-S=1 cost on GPU-resident experts. Revisit speculation only after that +lands. Until then, benchmark and serve runs on this machine should set +`DRAFT=0` explicitly: with a working MTP head present, the engine's +`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%. + +Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1), +`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2). + +## AVX-512 int4 kernel qualification (July 13) + +The candidate kernel from section "Candidate not yet merged" was put through +the three measurement gates it owed. The working-tree build exposes it behind +a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below +uses one binary. + +**Numeric error (passed, better than the incumbent).** A standalone harness +compared the AVX-512 dual-accumulator kernel and the engine's sequential +scalar-f32 order against a double-precision oracle over the real expert row +shapes (I=6144 and I=2048), 70k+ random rows across activation scales from +0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign +cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the +scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3, +RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less +rounding than sequential summation, so the "output divergence" is the new +kernel being closer to exact, not further. + +**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251 +continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off), +perplexity 5.99 vs 5.98 — a 0.24% difference with per-passage signs split +4/4. Statistically indistinguishable. + +**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts, +`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each: + +| Prompt | on | off | gain | +| --- | ---: | ---: | ---: | +| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% | +| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% | +| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% | + +The earlier +15% estimate came from a low 5.12 baseline and is retired. The +gain is workload-dependent for a structural reason: the kernel only +accelerates experts computed on CPU, and the water-cycle prompt's routing +happens to be served almost entirely from the GPU hot tier. This is the +clearest evidence yet that the next big lever is making GPU experts a +compute tier (Tensor Core grouped GEMM), which would also shrink every +CPU-side gain — and, by making multi-position verify batches cheap, +would re-open the MTP speculation door (69-79% acceptance is already +banked above). + +Verdict: the kernel is numerically superior, quality-neutral, and worth ++4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only +open item is the policy call that outputs are not bit-identical to the old +(worse) accumulation order. Logs: +`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`, +`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`. + +## Why six 5090s did not match two RTX PRO 6000s + +The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000 +Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32 +GB. Both machines may total 192 GB, but expert residency is constrained per +rank and runtime state is replicated. Six smaller memory islands are not +equivalent to two large ones. + +The 5090 has strong compute and is cost-effective for concurrency, but this +workload is dominated by resident expert capacity and miss recovery. Changing +TP/PP layout cannot repair a policy that replays an entire step after any +expert miss. + +## Conclusions for colibri + +The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with +different bandwidth and latency, so a model larger than VRAM can run. The +full-resident experiments strengthened it: on this machine the whole int4 +model fits in VRAM+RAM, and once disk left the decode path colibri at +6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s), +because colibri never replays a whole step on an expert miss. + +The remaining bottleneck is no longer storage or scheduling — it is the int4 +expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`), +with attention (25%) next in line. Priorities: + +1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity + evaluation, fixed greedy repeat runs) and take its ~15% if it passes. +2. Add finer counters to CPU expert execution (experts per layer, rows per + expert, per-matrix time) to guide kernel work. +3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never + mix sampled runs into comparisons. +4. Once the expert path is stably above about 7 tok/s, optimize attention + projection/RoPE and score-softmax-value. + +Twenty to thirty tokens per second has still not been demonstrated on this +machine; 6.84 tok/s (256-token greedy) is the current ceiling. + +## Last known lab state + +The fastest configuration overall is colibri full-resident in +`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the +`glm52-moet-real` service before benchmarking it and restore the service +afterwards). The fastest vLLM-Moet configuration was the TP4 service on port +8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle. +Preserve both pack directories until the PP experiment no longer needs to be +reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`. +These paths describe the lab machine and are not repository dependencies.