docs: GLM-5.2 6x RTX 5090 full-residency experiment — 6.84 tok/s, honest dead-ends log (#94)
Records the July 12-13 lab findings on one 6x RTX 5090 machine: - vLLM-Moet comparison and why per-rank expert residency dominates - colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the tested-and-rejected directions documented - MTP speculation: broken int4 head identified (issue #8), int8 head reaches 69-79% acceptance but MoE verify batches scale expert cost with S, so speculation loses at every depth; revisit after GPU grouped GEMM - AVX-512 int4 kernel qualification: numerically better than the old order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing README links the record from the honest-numbers section.
This commit is contained in:
ZacharyZcR
@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
|
||||
|
||||
## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
|
||||
|
||||
Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
|
||||
|
||||
| metric | value |
|
||||
|---|---|
|
||||
| model on disk (int4 container) | ~370 GB |
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
# GLM-5.2 on 6x RTX 5090
|
||||
|
||||
This note records the July 12, 2026 experiments that compared colibri with
|
||||
vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon
|
||||
Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe
|
||||
storage. It exists to preserve the measured facts and avoid repeating the same
|
||||
dead ends.
|
||||
|
||||
The day had two halves. The morning runs (below) measured colibri at 0.12
|
||||
tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on
|
||||
colibri (see "colibri full-resident experiments") removed disk from the decode
|
||||
path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark
|
||||
and 6.84 tok/s on a 256-token run — the fastest single-request result measured
|
||||
on this machine so far.
|
||||
|
||||
## Models and artifacts
|
||||
|
||||
- Official NVFP4 model: `/data/models/GLM-5.2-NVFP4` (434 GB, 47 shards)
|
||||
- colibri INT4 model: `/data/models/GLM-5.2-colibri-int4` (144 shards)
|
||||
- vLLM-Moet TP4 pack: `/data/models/GLM-5.2-NVFP4-packs` (190 GB)
|
||||
- vLLM-Moet TP2/PP3 pack: `/data/models/GLM-5.2-NVFP4-packs-tp2pp3` (190 GB)
|
||||
- vLLM-Moet checkout: `/data/test/vLLM-Moet`
|
||||
- Built image: `vllm-moet-sm120:v024`
|
||||
|
||||
The official model was downloaded through `hf-mirror.com` with
|
||||
`HF_HUB_DISABLE_XET=1`. Direct Hugging Face access failed and Xet/CAS stalled;
|
||||
ordinary HTTP from the mirror sustained roughly 22-46 MB/s.
|
||||
|
||||
## Kernel validation
|
||||
|
||||
All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke
|
||||
tests. Representative correctness results were:
|
||||
|
||||
| Path | Shape | Max relative error | Cosine similarity |
|
||||
| --- | --- | ---: | ---: |
|
||||
| Decode | T=2 | 2.086e-2 | 0.999878 |
|
||||
| NVFP4 delta | T=2 | 1.818e-2 | 0.999884 |
|
||||
| AFRAG prefill | T=128 | 2.591e-2 | 0.999883 |
|
||||
|
||||
The kernels work on SM120. The performance limitation below is therefore not
|
||||
an unsupported-GPU fallback.
|
||||
|
||||
## Real generation results
|
||||
|
||||
All decode rates below are for a single request, not aggregate throughput.
|
||||
|
||||
| Runtime and layout | GPU use | TTFT | Decode rate | Important observation |
|
||||
| --- | --- | ---: | ---: | --- |
|
||||
| colibri INT4 (morning, partial residency) | 6 GPUs | about 42 s | 0.12 tok/s | GPU hot tier was active, but dense compute remained on CPU |
|
||||
| colibri INT4 (afternoon, full residency) | 6 GPUs + 24 cores | — | 6.28-6.84 tok/s | all 19,456 experts resident in VRAM+RAM, zero disk waits |
|
||||
| vLLM-Moet TP4, 12 GiB expert cache/card | GPUs 0-3 | 2.03 s | median 2.5 tok/s | 25.3% coverage; 100% whole-step replay |
|
||||
| vLLM-Moet TP4, 14 GiB expert cache/card | GPUs 0-3 | not remeasured | median 2.6 tok/s | 29.5% coverage; replay still 100% |
|
||||
| vLLM-Moet TP2 x PP3 | all 6 GPUs | 5.39 s | 1.78 tok/s | more local coverage, but serial PP and communication made latency worse |
|
||||
|
||||
The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only
|
||||
about 0.1 tok/s because every decode step still missed experts and replayed the
|
||||
whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not
|
||||
reduce replay enough to offset pipeline and PYNCCL communication costs.
|
||||
|
||||
The morning colibri run generated 16 tokens in 132.46 seconds. Its measured
|
||||
hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the
|
||||
expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk
|
||||
reads (29.31 s), and attention (14.28 s). The afternoon experiments below
|
||||
showed the dominant problem was expert loading from disk, not the CUDA path
|
||||
itself.
|
||||
|
||||
## colibri full-resident experiments (afternoon)
|
||||
|
||||
Test directory: `/data/test/colibri-full-resident`. Related PR:
|
||||
[#80](https://github.com/JustVugg/colibri/pull/80) (on top of PR #68). All A/B
|
||||
runs used the same binary flags, environment, prompt, `TEMP=0`, and token
|
||||
count; only decode throughput is compared, never load or prefill time.
|
||||
|
||||
### The optimization ladder (each step cumulative)
|
||||
|
||||
| Change | Decode rate | Evidence |
|
||||
| --- | ---: | --- |
|
||||
| Baseline: 150 GB VRAM + 150 GB RAM fixed placement | 2.30 tok/s | 4.15 s of disk waits per 20 tokens |
|
||||
| Full residency: all 19,456 experts in VRAM+RAM | 5.77 tok/s | 100% expert hit, zero disk waits |
|
||||
| In-session dynamic GPU expert repinning (`REPIN=16`) | 6.00 tok/s | expert compute 6.76 s → 5.96 s |
|
||||
| 24 physical cores, `OMP_PROC_BIND=spread OMP_PLACES=cores` | +39.6% vs unpinned | unpinned 3.64 → pinned 5.08 tok/s on the same prompt |
|
||||
| Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT) | 6.05-6.08 tok/s | GPU expert calls 36,865 → 37,285 |
|
||||
| Decode swap cap 32 → 16 per repin round | 6.10-6.28 tok/s | swap cost 0.18 → 0.09 s/round |
|
||||
|
||||
The final configuration measured 6.28 tok/s on the fixed 96-token greedy
|
||||
benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed
|
||||
costs; always state the token count when quoting a rate).
|
||||
|
||||
Winning layout, found automatically:
|
||||
|
||||
```text
|
||||
GPU experts: 9,343 / 19,456 (176.73 GB across six cards)
|
||||
RAM experts: 10,113 / 19,456 (about 191.3 GB)
|
||||
Disk service/wait during decode: 0 s
|
||||
```
|
||||
|
||||
Run configuration: `COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto
|
||||
CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto`, with
|
||||
`OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores` (now the default
|
||||
generated by `resource_plan.py`).
|
||||
|
||||
### Directions tested and rejected (all reverted)
|
||||
|
||||
- Faster (`REPIN=8`) or slower (`REPIN=32`) repinning, and 64 swaps per
|
||||
round: swap/sync overhead or stale GPU expert sets lose more than they gain.
|
||||
16 tokens between repins and 16 swaps per round is the current local optimum
|
||||
(8 swaps also regresses: fewer GPU calls outweigh the sync savings).
|
||||
- A second prefill correction pass: overfits the short prompt's routing and
|
||||
disturbs the history-based layout (5.86 tok/s).
|
||||
- Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery
|
||||
(6.15 tok/s): both cost as much as re-reading from page cache.
|
||||
- CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30
|
||||
tok/s); GPU expert groups are too small a fraction of step time, and the
|
||||
extra threads compete with the 24 CPU workers.
|
||||
- OpenMP restructuring of RAM experts (row batching across experts,
|
||||
persistent teams): neutral or negative; libgomp already reuses hot teams,
|
||||
and interleaving experts in one static schedule destroys per-expert
|
||||
NUMA/cache locality.
|
||||
- `numactl --interleave=all`, 2 MB transparent huge pages, 12-core runs,
|
||||
smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI
|
||||
int4×int8 decode: all neutral or negative. Capacity and TLB are not the
|
||||
limit; raw history counts predict routing better than normalized ones.
|
||||
- Next-layer expert prediction with GPU staging: the routing signal is real
|
||||
(70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the
|
||||
expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth
|
||||
revisiting only with dedicated streams that do not stall attention.
|
||||
|
||||
### Candidate not yet merged: AVX-512 int4 kernel
|
||||
|
||||
The lossless int4→float RAM expert kernel still uses 256-bit AVX2. A 512-bit
|
||||
unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 → 5.89
|
||||
tok/s, expert time −19.7%), and a dual-accumulator variant added a little more
|
||||
(5.89 → 5.94 tok/s at 96 tokens). However, the changed accumulation order
|
||||
produces small float differences and the 64-token outputs eventually diverge,
|
||||
so it cannot be claimed bit-exact. It stays out of the tree until it has
|
||||
numeric-error unit tests, a perplexity/token-consistency evaluation, and an
|
||||
explicit decision on whether float-accumulation differences are acceptable.
|
||||
Row blocking (2/4 output rows) looked good in short microbenchmarks but the
|
||||
gain disappeared over 10,000 sustained iterations and in a 256-token run
|
||||
(6.84 vs 6.83 tok/s) — a turbo/cache artifact, reverted.
|
||||
|
||||
### Where the time goes now
|
||||
|
||||
At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other
|
||||
2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of
|
||||
CPU time in `matmul_qt` and 8.8% in `rope_interleave`; a per-thread RoPE
|
||||
sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention
|
||||
−5.0%) with no math change. Scheduling-level CPU optimizations are exhausted —
|
||||
the remaining expert time is the int4 matmul kernel itself.
|
||||
|
||||
## MTP speculative decoding at full residency (July 13)
|
||||
|
||||
Motivation: with all experts resident and disk waits at zero, the old
|
||||
objection to speculation (a rejected draft wastes a disk read) no longer
|
||||
applies, so the July 3 negative result deserved a retest. All runs used the
|
||||
same binary and environment as the 16-swap configuration above, `TEMP=0`,
|
||||
96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy
|
||||
output degenerates into repetition, and a non-degenerate "water cycle"
|
||||
prompt).
|
||||
|
||||
Phase 1 found the installed MTP head unusable: acceptance 0-4% at every
|
||||
depth, exactly the known int4-head defect from issue #8 (the three
|
||||
`out-mtp-*.safetensors` byte sizes matched the "unusable" fingerprint).
|
||||
Speculation with the broken head costs 15-18% throughput — pure verification
|
||||
tax.
|
||||
|
||||
Phase 2 replaced the head with the community int8 MTP shards
|
||||
(`mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp`, ~10 GB; originals kept as
|
||||
`.int4.bak`). The head then worked as designed — and speculation still lost
|
||||
at every depth:
|
||||
|
||||
| Prompt | DRAFT | tok/s | tok/forward | Acceptance |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| water cycle | 0 | 6.79 | 1.01 | — |
|
||||
| water cycle | 1 | 6.45 | 1.81 | 79% |
|
||||
| water cycle | 2 | 5.65 | 2.29 | 64% |
|
||||
| water cycle | 3 | 6.13 | 3.10 | 69% |
|
||||
| sky (degenerate) | 0 | 6.12 | 1.01 | — |
|
||||
| sky (degenerate) | 1 | 5.51 | 1.75 | 73% |
|
||||
| sky (degenerate) | 2 | 4.42 | 1.88 | 44% |
|
||||
| sky (degenerate) | 3 | 3.86 / 3.81 | 2.13 | 38% |
|
||||
|
||||
The failure is structural, not a tuning problem. In a dense model a verify
|
||||
batch of S tokens rereads the same weights once, so speculation amortizes
|
||||
the dominant cost. In this MoE the dominant cost is per-unique-expert weight
|
||||
processing, and the S positions of a verify batch route to mostly different
|
||||
experts: measured per-forward expert time scaled almost linearly with S
|
||||
(80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt).
|
||||
There is no amortization to collect, and the MTP head's own forward (int8
|
||||
experts, one extra layer per draft token) adds cost on top. Even 79%
|
||||
acceptance at DRAFT=1 nets -5%.
|
||||
|
||||
Two useful facts survive the negative result. First, the int8 head is
|
||||
genuinely good: 69-79% chained top-1 acceptance on clean text (the int8
|
||||
shards stay installed; the int4 originals were defective). Second, the
|
||||
conclusion inverts if routed-expert compute per extra position ever becomes
|
||||
cheap — e.g. Tensor Core grouped GEMM computing an S=4 verify batch at
|
||||
near-S=1 cost on GPU-resident experts. Revisit speculation only after that
|
||||
lands. Until then, benchmark and serve runs on this machine should set
|
||||
`DRAFT=0` explicitly: with a working MTP head present, the engine's
|
||||
`DRAFT=-1` default auto-enables 3 drafts and silently costs 10-37%.
|
||||
|
||||
Logs: `/data/test/colibri-six-gpu-ab-mtp-*-20260712.log` (phase 1),
|
||||
`/data/test/colibri-six-gpu-ab-mtp8-*-20260713.log` (phase 2).
|
||||
|
||||
## AVX-512 int4 kernel qualification (July 13)
|
||||
|
||||
The candidate kernel from section "Candidate not yet merged" was put through
|
||||
the three measurement gates it owed. The working-tree build exposes it behind
|
||||
a runtime switch (`I4_ACC512=0/1`, default on), so every comparison below
|
||||
uses one binary.
|
||||
|
||||
**Numeric error (passed, better than the incumbent).** A standalone harness
|
||||
compared the AVX-512 dual-accumulator kernel and the engine's sequential
|
||||
scalar-f32 order against a double-precision oracle over the real expert row
|
||||
shapes (I=6144 and I=2048), 70k+ random rows across activation scales from
|
||||
0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign
|
||||
cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the
|
||||
scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3,
|
||||
RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less
|
||||
rounding than sequential summation, so the "output divergence" is the new
|
||||
kernel being closer to exact, not further.
|
||||
|
||||
**Quality (passed).** SCORE log-likelihood over 8 multi-domain passages (251
|
||||
continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off),
|
||||
perplexity 5.99 vs 5.98 — a 0.24% difference with per-passage signs split
|
||||
4/4. Statistically indistinguishable.
|
||||
|
||||
**Throughput (+4% mean, +7% where it applies).** Three fixed greedy prompts,
|
||||
`TEMP=0 DRAFT=0`, 96 tokens, ABAB-interleaved on/off, two repeats each:
|
||||
|
||||
| Prompt | on | off | gain |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| sky | 6.08 / 6.08 | 5.75 / 5.62 | +7.0% |
|
||||
| water cycle | 6.51 / 6.50 | 6.53 / 6.52 | 0% |
|
||||
| rivers | 5.79 / 5.82 | 5.24 / 5.57 | +7.4% |
|
||||
|
||||
The earlier +15% estimate came from a low 5.12 baseline and is retired. The
|
||||
gain is workload-dependent for a structural reason: the kernel only
|
||||
accelerates experts computed on CPU, and the water-cycle prompt's routing
|
||||
happens to be served almost entirely from the GPU hot tier. This is the
|
||||
clearest evidence yet that the next big lever is making GPU experts a
|
||||
compute tier (Tensor Core grouped GEMM), which would also shrink every
|
||||
CPU-side gain — and, by making multi-position verify batches cheap,
|
||||
would re-open the MTP speculation door (69-79% acceptance is already
|
||||
banked above).
|
||||
|
||||
Verdict: the kernel is numerically superior, quality-neutral, and worth
|
||||
+4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only
|
||||
open item is the policy call that outputs are not bit-identical to the old
|
||||
(worse) accumulation order. Logs:
|
||||
`/data/test/colibri-six-gpu-ab-avx512q-*-20260713.log`,
|
||||
`/data/test/score-acc{0,1}.out`, harness `/data/test/i4_numeric_test.c`.
|
||||
|
||||
## Why six 5090s did not match two RTX PRO 6000s
|
||||
|
||||
The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000
|
||||
Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32
|
||||
GB. Both machines may total 192 GB, but expert residency is constrained per
|
||||
rank and runtime state is replicated. Six smaller memory islands are not
|
||||
equivalent to two large ones.
|
||||
|
||||
The 5090 has strong compute and is cost-effective for concurrency, but this
|
||||
workload is dominated by resident expert capacity and miss recovery. Changing
|
||||
TP/PP layout cannot repair a policy that replays an entire step after any
|
||||
expert miss.
|
||||
|
||||
## Conclusions for colibri
|
||||
|
||||
The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with
|
||||
different bandwidth and latency, so a model larger than VRAM can run. The
|
||||
full-resident experiments strengthened it: on this machine the whole int4
|
||||
model fits in VRAM+RAM, and once disk left the decode path colibri at
|
||||
6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s),
|
||||
because colibri never replays a whole step on an expert miss.
|
||||
|
||||
The remaining bottleneck is no longer storage or scheduling — it is the int4
|
||||
expert matmul on CPU (56% of step time, 84.9% of CPU samples in `matmul_qt`),
|
||||
with attention (25%) next in line. Priorities:
|
||||
|
||||
1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity
|
||||
evaluation, fixed greedy repeat runs) and take its ~15% if it passes.
|
||||
2. Add finer counters to CPU expert execution (experts per layer, rows per
|
||||
expert, per-matrix time) to guide kernel work.
|
||||
3. Keep all A/B tests on fixed `TEMP=0` prompts and fixed token counts; never
|
||||
mix sampled runs into comparisons.
|
||||
4. Once the expert path is stably above about 7 tok/s, optimize attention
|
||||
projection/RoPE and score-softmax-value.
|
||||
|
||||
Twenty to thirty tokens per second has still not been demonstrated on this
|
||||
machine; 6.84 tok/s (256-token greedy) is the current ceiling.
|
||||
|
||||
## Last known lab state
|
||||
|
||||
The fastest configuration overall is colibri full-resident in
|
||||
`/data/test/colibri-full-resident` (6.28-6.84 tok/s; stop the
|
||||
`glm52-moet-real` service before benchmarking it and restore the service
|
||||
afterwards). The fastest vLLM-Moet configuration was the TP4 service on port
|
||||
8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle.
|
||||
Preserve both pack directories until the PP experiment no longer needs to be
|
||||
reproduced. Raw A/B logs live under `/data/test/colibri-six-gpu-ab-*.log`.
|
||||
These paths describe the lab machine and are not repository dependencies.
|
||||
Reference in New Issue
Block a user