danielkrause/colibri-strix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: GLM-5.2 6x RTX 5090 full-residency experiment — 6.84 tok/s, honest dead-ends log (#94)

Browse Source 
Records the July 12-13 lab findings on one 6x RTX 5090 machine:
- vLLM-Moet comparison and why per-rank expert residency dominates
- colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the
  tested-and-rejected directions documented
- MTP speculation: broken int4 head identified (issue #8), int8 head
  reaches 69-79% acceptance but MoE verify batches scale expert cost
  with S, so speculation loses at every depth; revisit after GPU
  grouped GEMM
- AVX-512 int4 kernel qualification: numerically better than the old
  order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing

README links the record from the honest-numbers section.
This commit is contained in:
ZacharyZcR
2026-07-13 04:06:51 +08:00
committed by GitHub
parent f38e74efe7
commit 3fb5a00106
2 changed files with 304 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2
View File
@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
| metric | value |
|---|---|
| model on disk (int4 container) | ~370 GB |
docs/experiments/glm52-6x5090-2026-07-12.md
+302
View File
@@ -0,0 +1,302 @@
# GLM-5.2 on 6x RTX 5090
This note records the July 12, 2026 experiments that compared colibri with
vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon
Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe
storage. It exists to preserve the measured facts and avoid repeating the same
dead ends.
The day had two halves. The morning runs (below) measured colibri at 0.12
tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on
colibri (see "colibri full-resident experiments") removed disk from the decode
path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark
and 6.84 tok/s on a 256-token run — the fastest single-request result measured
on this machine so far.
## Models and artifacts
- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards)
- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards)
- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB)
- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB)
- vLLM-Moet checkout: `/data/test/vLLM-Moet`
- Built image: `vllm-moet-sm120:v024`
The official model was downloaded through `hf-mirror.com` with
`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled;
ordinary HTTP from the mirror sustained roughly 22-46 MB/s.
## Kernel validation
All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke
tests. Representative correctness results were:
| Path | Shape | Max relative error | Cosine similarity |
| --- | --- | ---: | ---: |
| Decode | T=2 | 2.086e-2 | 0.999878 |
| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 |
| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 |
The kernels work on SM120. The performance limitation below is therefore not
an unsupported-GPU fallback.
## Real generation results
All decode rates below are for a single request, not aggregate throughput.
| Runtime and layout | GPU use | TTFT | Decode rate | Important observation |
| --- | --- | ---: | ---: | --- |
| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU |
| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | — | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits |
| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay |
| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% |
| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse |
The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only
about 0.1 tok/s because every decode step still missed experts and replayed the
whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not
reduce replay enough to offset pipeline and PYNCCL communication costs.
The morning colibri run generated 16 tokens in 132.46 seconds. Its measured
hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the
expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk
reads (29.31 s), and attention (14.28 s). The afternoon experiments below
showed the dominant problem was expert loading from disk, not the CUDA path
itself.
## colibri full-resident experiments (afternoon)
Test directory: `/data/test/colibri-full-resident`. Related PR:
[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B
runs used the same binary flags, environment, prompt, `TEMP=0`, and token
count; only decode throughput is compared, never load or prefill time.
### The optimization ladder (each step cumulative)
| Change | Decode rate | Evidence |
| --- | ---: | --- |
| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens |
| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits |
| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s → 5.96 s |
| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 → pinned 5.08 tok/s on the same prompt |
| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 → 37,285 |
| Decode swap cap 32 → 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 → 0.09 s/round |
The final configuration measured 6.28 tok/s on the fixed 96-token greedy
benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed
costs; always state the token count when quoting a rate).
Winning layout, found automatically:
```text
GPU experts: 9,343 / 19,456 (176.73 GB across six cards)
RAM experts: 10,113 / 19,456 (about 191.3 GB)
Disk service/wait during decode: 0 s
```
Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto
CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with
`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default
generated by `resource_plan.py`).
### Directions tested and rejected (all reverted)
- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per
round: swap/sync overhead or stale GPU expert sets lose more than they gain.
16 tokens between repins and 16 swaps per round is the current local optimum
(8 swaps also regresses: fewer GPU calls outweigh the sync savings).
- A second prefill correction pass: overfits the short prompt's routing and
disturbs the history-based layout (5.86 tok/s).
- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery
(6.15 tok/s): both cost as much as re-reading from page cache.
- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30
tok/s); GPU expert groups are too small a fraction of step time, and the
extra threads compete with the 24 CPU workers.
- OpenMP restructuring of RAM experts (row batching across experts,
persistent teams): neutral or negative; libgomp already reuses hot teams,
and interleaving experts in one static schedule destroys per-expert
NUMA/cache locality.
- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs,
smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI
int4×int8 decode: all neutral or negative. Capacity and TLB are not the
limit; raw history counts predict routing better than normalized ones.
- Next-layer expert prediction with GPU staging: the routing signal is real
(70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the
expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth
revisiting only with dedicated streams that do not stall attention.
### Candidate not yet merged: AVX-512 int4 kernel
The lossless int4→float RAM expert kernel still uses 256-bit AVX2. A 512-bit
unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 → 5.89
tok/s, expert time −19.7%), and a dual-accumulator variant added a little more
(5.89 → 5.94 tok/s at 96 tokens). However, the changed accumulation order
produces small float differences and the 64-token outputs eventually diverge,
so it cannot be claimed bit-exact. It stays out of the tree until it has
numeric-error unit tests, a perplexity/token-consistency evaluation, and an
explicit decision on whether float-accumulation differences are acceptable.
Row blocking (2/4 output rows) looked good in short microbenchmarks but the
gain disappeared over 10,000 sustained iterations and in a 256-token run
(6.84 vs 6.83 tok/s) — a turbo/cache artifact, reverted.
### Where the time goes now
At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other
2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of
CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE
sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention
−5.0%) with no math change. Scheduling-level CPU optimizations are exhausted —
the remaining expert time is the int4 matmul kernel itself.
## MTP speculative decoding at full residency (July 13)
Motivation: with all experts resident and disk waits at zero, the old
objection to speculation (a rejected draft wastes a disk read) no longer
applies, so the July 3 negative result deserved a retest. All runs used the
same binary and environment as the 16-swap configuration above, `TEMP=0`,
96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy
output degenerates into repetition, and a non-degenerate "water cycle"
prompt).
Phase 1 found the installed MTP head unusable: acceptance 0-4% at every
depth, exactly the known int4-head defect from issue #8 (the three
`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint).
Speculation with the broken head costs 15-18% throughput — pure verification
tax.
Phase 2 replaced the head with the community int8 MTP shards
(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as
`.int4.bak`). The head then worked as designed — and speculation still lost
at every depth:
| Prompt | DRAFT | tok/s | tok/forward | Acceptance |
| --- | ---: | ---: | ---: | ---: |
| water cycle | 0 | 6.79 | 1.01 | — |
| water cycle | 1 | 6.45 | 1.81 | 79% |
| water cycle | 2 | 5.65 | 2.29 | 64% |
| water cycle | 3 | 6.13 | 3.10 | 69% |
| sky (degenerate) | 0 | 6.12 | 1.01 | — |
| sky (degenerate) | 1 | 5.51 | 1.75 | 73% |
| sky (degenerate) | 2 | 4.42 | 1.88 | 44% |
| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% |
The failure is structural, not a tuning problem. In a dense model a verify
batch of S tokens rereads the same weights once, so speculation amortizes
the dominant cost. In this MoE the dominant cost is per-unique-expert weight
processing, and the S positions of a verify batch route to mostly different
experts: measured per-forward expert time scaled almost linearly with S
(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt).
There is no amortization to collect, and the MTP head's own forward (int8
experts, one extra layer per draft token) adds cost on top. Even 79%
acceptance at DRAFT=1 nets -5%.
Two useful facts survive the negative result. First, the int8 head is
genuinely good: 69-79% chained top-1 acceptance on clean text (the int8
shards stay installed; the int4 originals were defective). Second, the
conclusion inverts if routed-expert compute per extra position ever becomes
cheap — e.g. Tensor Core grouped GEMM computing an S=4 verify batch at
near-S=1 cost on GPU-resident experts. Revisit speculation only after that
lands. Until then, benchmark and serve runs on this machine should set
`DRAFT=0` explicitly: with a working MTP head present, the engine's
`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%.
Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1),
`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2).
## AVX-512 int4 kernel qualification (July 13)
The candidate kernel from section "Candidate not yet merged" was put through
the three measurement gates it owed. The working-tree build exposes it behind
a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below
uses one binary.
**Numeric error (passed, better than the incumbent).** A standalone harness
compared the AVX-512 dual-accumulator kernel and the engine's sequential
scalar-f32 order against a double-precision oracle over the real expert row
shapes (I=6144 and I=2048), 70k+ random rows across activation scales from
0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign
cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the
scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3,
RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less
rounding than sequential summation, so the "output divergence" is the new
kernel being closer to exact, not further.
**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251
continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off),
perplexity 5.99 vs 5.98 — a 0.24% difference with per-passage signs split
4/4. Statistically indistinguishable.
**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts,
`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each:
| Prompt | on | off | gain |
| --- | ---: | ---: | ---: |
| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% |
| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% |
| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% |
The earlier +15% estimate came from a low 5.12 baseline and is retired. The
gain is workload-dependent for a structural reason: the kernel only
accelerates experts computed on CPU, and the water-cycle prompt's routing
happens to be served almost entirely from the GPU hot tier. This is the
clearest evidence yet that the next big lever is making GPU experts a
compute tier (Tensor Core grouped GEMM), which would also shrink every
CPU-side gain — and, by making multi-position verify batches cheap,
would re-open the MTP speculation door (69-79% acceptance is already
banked above).
Verdict: the kernel is numerically superior, quality-neutral, and worth
+4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only
open item is the policy call that outputs are not bit-identical to the old
(worse) accumulation order. Logs:
`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`,
`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`.
## Why six 5090s did not match two RTX PRO 6000s
The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000
Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32
GB. Both machines may total 192 GB, but expert residency is constrained per
rank and runtime state is replicated. Six smaller memory islands are not
equivalent to two large ones.
The 5090 has strong compute and is cost-effective for concurrency, but this
workload is dominated by resident expert capacity and miss recovery. Changing
TP/PP layout cannot repair a policy that replays an entire step after any
expert miss.
## Conclusions for colibri
The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with
different bandwidth and latency, so a model larger than VRAM can run. The
full-resident experiments strengthened it: on this machine the whole int4
model fits in VRAM+RAM, and once disk left the decode path colibri at
6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s),
because colibri never replays a whole step on an expert miss.
The remaining bottleneck is no longer storage or scheduling — it is the int4
expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`),
with attention (25%) next in line. Priorities:
1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity
evaluation, fixed greedy repeat runs) and take its ~15% if it passes.
2. Add finer counters to CPU expert execution (experts per layer, rows per
expert, per-matrix time) to guide kernel work.
3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never
mix sampled runs into comparisons.
4. Once the expert path is stably above about 7 tok/s, optimize attention
projection/RoPE and score-softmax-value.
Twenty to thirty tokens per second has still not been demonstrated on this
machine; 6.84 tok/s (256-token greedy) is the current ceiling.
## Last known lab state
The fastest configuration overall is colibri full-resident in
`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the
`glm52-moet-real` service before benchmarking it and restore the service
afterwards). The fastest vLLM-Moet configuration was the TP4 service on port
8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle.
Preserve both pack directories until the PP experiment no longer needs to be
reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`.
These paths describe the lab machine and are not repository dependencies.