colibri-strix/docs/experiments/glm52-6x5090-2026-07-12.md

GLM-5.2 on 6x RTX 5090

This note records the July 12, 2026 experiments that compared colibri with vLLM-Moet on one machine with six RTX 5090 GPUs (32 GiB each), dual Intel Xeon Silver 4510 (24 physical cores / 48 threads), 251 GiB of RAM, and local NVMe storage. It exists to preserve the measured facts and avoid repeating the same dead ends.

The day had two halves. The morning runs (below) measured colibri at 0.12 tok/s and vLLM-Moet at 2.3-2.7 tok/s. The afternoon full-resident work on colibri (see "colibri full-resident experiments") removed disk from the decode path entirely and reached 6.28 tok/s on the fixed 96-token greedy benchmark and 6.84 tok/s on a 256-token run — the fastest single-request result measured on this machine so far.

Models and artifacts

• Official NVFP4 model: /data/models/GLM-5.2-NVFP4 (434 GB, 47 shards)
• colibri INT4 model: /data/models/GLM-5.2-colibri-int4 (144 shards)
• vLLM-Moet TP4 pack: /data/models/GLM-5.2-NVFP4-packs (190 GB)
• vLLM-Moet TP2/PP3 pack: /data/models/GLM-5.2-NVFP4-packs-tp2pp3 (190 GB)
• vLLM-Moet checkout: /data/test/vLLM-Moet
• Built image: vllm-moet-sm120:v024

The official model was downloaded through hf-mirror.com with HF_HUB_DISABLE_XET=1. Direct Hugging Face access failed and Xet/CAS stalled; ordinary HTTP from the mirror sustained roughly 22-46 MB/s.

Kernel validation

All six RTX 5090 cards passed the real 2-bit and NVFP4 delta kernel smoke tests. Representative correctness results were:

Path	Shape	Max relative error	Cosine similarity
Decode	T=2	2.086e-2	0.999878
NVFP4 delta	T=2	1.818e-2	0.999884
AFRAG prefill	T=128	2.591e-2	0.999883

The kernels work on SM120. The performance limitation below is therefore not an unsupported-GPU fallback.

Real generation results

All decode rates below are for a single request, not aggregate throughput.

Runtime and layout	GPU use	TTFT	Decode rate	Important observation
colibri INT4 (morning, partial residency)	6 GPUs	about 42 s	0.12 tok/s	GPU hot tier was active, but dense compute remained on CPU
colibri INT4 (afternoon, full residency)	6 GPUs + 24 cores	—	6.28-6.84 tok/s	all 19,456 experts resident in VRAM+RAM, zero disk waits
vLLM-Moet TP4, 12 GiB expert cache/card	GPUs 0-3	2.03 s	median 2.5 tok/s	25.3% coverage; 100% whole-step replay
vLLM-Moet TP4, 14 GiB expert cache/card	GPUs 0-3	not remeasured	median 2.6 tok/s	29.5% coverage; replay still 100%
vLLM-Moet TP2 x PP3	all 6 GPUs	5.39 s	1.78 tok/s	more local coverage, but serial PP and communication made latency worse

The TP4 runs measured 2.3-2.7 tok/s. Raising the expert cache gained only about 0.1 tok/s because every decode step still missed experts and replayed the whole step. TP2/PP3 raised local expert coverage to roughly 55-62%, but did not reduce replay enough to offset pipeline and PYNCCL communication costs.

The morning colibri run generated 16 tokens in 132.46 seconds. Its measured hot-expert tier was 77.48 GB across six GPUs, RAM peaked at 175.94 GB, and the expert hit rate was 71%. The largest costs were expert matmul (81.37 s), disk reads (29.31 s), and attention (14.28 s). The afternoon experiments below showed the dominant problem was expert loading from disk, not the CUDA path itself.

colibri full-resident experiments (afternoon)

Test directory: /data/test/colibri-full-resident. Related PR: #80 (on top of PR #68). All A/B runs used the same binary flags, environment, prompt, TEMP=0, and token count; only decode throughput is compared, never load or prefill time.

The optimization ladder (each step cumulative)

Change	Decode rate	Evidence
Baseline: 150 GB VRAM + 150 GB RAM fixed placement	2.30 tok/s	4.15 s of disk waits per 20 tokens
Full residency: all 19,456 experts in VRAM+RAM	5.77 tok/s	100% expert hit, zero disk waits
In-session dynamic GPU expert repinning (REPIN=16)	6.00 tok/s	expert compute 6.76 s → 5.96 s
24 physical cores, OMP_PROC_BIND=spread OMP_PLACES=cores	+39.6% vs unpinned	unpinned 3.64 → pinned 5.08 tok/s on the same prompt
Prefill corrects all 75 MoE layers once (454 ms, counted in TTFT)	6.05-6.08 tok/s	GPU expert calls 36,865 → 37,285
Decode swap cap 32 → 16 per repin round	6.10-6.28 tok/s	swap cost 0.18 → 0.09 s/round

The final configuration measured 6.28 tok/s on the fixed 96-token greedy benchmark and 6.84 tok/s on a 256-token run (longer decode amortizes fixed costs; always state the token count when quoting a rate).

Winning layout, found automatically:

GPU experts:  9,343 / 19,456  (176.73 GB across six cards)
RAM experts: 10,113 / 19,456  (about 191.3 GB)
Disk service/wait during decode: 0 s

Run configuration: COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=auto CUDA_DENSE=1 COLI_CUDA_ATTN=1 PIN_GB=all RAM_GB=auto, with OMP_NUM_THREADS=24 OMP_PROC_BIND=spread OMP_PLACES=cores (now the default generated by resource_plan.py).

Directions tested and rejected (all reverted)

• Faster (REPIN=8) or slower (REPIN=32) repinning, and 64 swaps per round: swap/sync overhead or stale GPU expert sets lose more than they gain. 16 tokens between repins and 16 swaps per round is the current local optimum (8 swaps also regresses: fewer GPU calls outweigh the sync savings).
• A second prefill correction pass: overfits the short prompt's routing and disturbs the history-based layout (5.86 tok/s).
• Lazy demotion of evicted GPU experts (4.77 tok/s) and GPU→RAM D2H recovery (6.15 tok/s): both cost as much as re-reading from page cache.
• CPU/GPU expert overlap via extra pthreads: no stable gain (6.09-6.30 tok/s); GPU expert groups are too small a fraction of step time, and the extra threads compete with the 24 CPU workers.
• OpenMP restructuring of RAM experts (row batching across experts, persistent teams): neutral or negative; libgomp already reuses hot teams, and interleaving experts in one static schedule destroys per-expert NUMA/cache locality.
• numactl --interleave=all, 2 MB transparent huge pages, 12-core runs, smaller VRAM reserve, per-layer-normalized static expert profiles, VNNI int4×int8 decode: all neutral or negative. Capacity and TLB are not the limit; raw history counts predict routing better than normalized ones.
• Next-layer expert prediction with GPU staging: the routing signal is real (70.6-78.9% recall, +7.8% GPU coverage) but PCIe staging contends with the expert/attention streams and loses end to end (5.39-5.44 tok/s). Worth revisiting only with dedicated streams that do not stall attention.

Candidate not yet merged: AVX-512 int4 kernel

The lossless int4→float RAM expert kernel still uses 256-bit AVX2. A 512-bit unpack+FMA version measured +15.0% throughput in a greedy A/B (5.12 → 5.89 tok/s, expert time −19.7%), and a dual-accumulator variant added a little more (5.89 → 5.94 tok/s at 96 tokens). However, the changed accumulation order produces small float differences and the 64-token outputs eventually diverge, so it cannot be claimed bit-exact. It stays out of the tree until it has numeric-error unit tests, a perplexity/token-consistency evaluation, and an explicit decision on whether float-accumulation differences are acceptable. Row blocking (2/4 output rows) looked good in short microbenchmarks but the gain disappeared over 10,000 sustained iterations and in a 256-token run (6.84 vs 6.83 tok/s) — a turbo/cache artifact, reverted.

Where the time goes now

At 6.00 tok/s: expert compute 5.96 s (56%), attention 2.62 s (25%), other 2.09 s (19%), disk 0 s. A gprof sample of a 96-token greedy run puts 84.9% of CPU time in matmul_qt and 8.8% in rope_interleave; a per-thread RoPE sin/cos cache for repeated positions cut projection/RoPE by 13.2% (attention −5.0%) with no math change. Scheduling-level CPU optimizations are exhausted — the remaining expert time is the int4 matmul kernel itself.

MTP speculative decoding at full residency (July 13)

Motivation: with all experts resident and disk waits at zero, the old objection to speculation (a rejected draft wastes a disk read) no longer applies, so the July 3 negative result deserved a retest. All runs used the same binary and environment as the 16-swap configuration above, TEMP=0, 96 new tokens, two prompts (the standard "sky is blue" prompt, whose greedy output degenerates into repetition, and a non-degenerate "water cycle" prompt).

Phase 1 found the installed MTP head unusable: acceptance 0-4% at every depth, exactly the known int4-head defect from issue #8 (the three out-mtp-*.safetensors byte sizes matched the "unusable" fingerprint). Speculation with the broken head costs 15-18% throughput — pure verification tax.

Phase 2 replaced the head with the community int8 MTP shards (mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp, ~10 GB; originals kept as .int4.bak). The head then worked as designed — and speculation still lost at every depth:

Prompt	DRAFT	tok/s	tok/forward	Acceptance
water cycle	0	6.79	1.01	—
water cycle	1	6.45	1.81	79%
water cycle	2	5.65	2.29	64%
water cycle	3	6.13	3.10	69%
sky (degenerate)	0	6.12	1.01	—
sky (degenerate)	1	5.51	1.75	73%
sky (degenerate)	2	4.42	1.88	44%
sky (degenerate)	3	3.86 / 3.81	2.13	38%

The failure is structural, not a tuning problem. In a dense model a verify batch of S tokens rereads the same weights once, so speculation amortizes the dominant cost. In this MoE the dominant cost is per-unique-expert weight processing, and the S positions of a verify batch route to mostly different experts: measured per-forward expert time scaled almost linearly with S (80 ms at S=1, 168 ms at S=2, 306 ms at S=4 on the water-cycle prompt). There is no amortization to collect, and the MTP head's own forward (int8 experts, one extra layer per draft token) adds cost on top. Even 79% acceptance at DRAFT=1 nets -5%.

Two useful facts survive the negative result. First, the int8 head is genuinely good: 69-79% chained top-1 acceptance on clean text (the int8 shards stay installed; the int4 originals were defective). Second, the conclusion inverts if routed-expert compute per extra position ever becomes cheap — e.g. Tensor Core grouped GEMM computing an S=4 verify batch at near-S=1 cost on GPU-resident experts. Revisit speculation only after that lands. Until then, benchmark and serve runs on this machine should set DRAFT=0 explicitly: with a working MTP head present, the engine's DRAFT=-1 default auto-enables 3 drafts and silently costs 10-37%.

Logs: /data/test/colibri-six-gpu-ab-mtp-*-20260712.log (phase 1), /data/test/colibri-six-gpu-ab-mtp8-*-20260713.log (phase 2).

AVX-512 int4 kernel qualification (July 13)

The candidate kernel from section "Candidate not yet merged" was put through the three measurement gates it owed. The working-tree build exposes it behind a runtime switch (I4_ACC512=0/1, default on), so every comparison below uses one binary.

Numeric error (passed, better than the incumbent). A standalone harness compared the AVX-512 dual-accumulator kernel and the engine's sequential scalar-f32 order against a double-precision oracle over the real expert row shapes (I=6144 and I=2048), 70k+ random rows across activation scales from 0.02 to 30, a non-multiple-of-32 tail case, and an alternating-sign cancellation stress. The AVX-512 kernel's error was 2-4x LOWER than the scalar order in every regime (e.g. gate/up rows: max rel 2.6e-4 vs 1.1e-3, RMS 5.7e-6 vs 1.9e-5); no non-finite values. Tree reduction accumulates less rounding than sequential summation, so the "output divergence" is the new kernel being closer to exact, not further.

Quality (passed). SCORE log-likelihood over 8 multi-domain passages (251 continuation tokens): total logprob -449.30 (AVX-512 on) vs -448.70 (off), perplexity 5.99 vs 5.98 — a 0.24% difference with per-passage signs split 4/4. Statistically indistinguishable.

Throughput (+4% mean, +7% where it applies). Three fixed greedy prompts, TEMP=0 DRAFT=0, 96 tokens, ABAB-interleaved on/off, two repeats each:

Prompt	on	off	gain
sky	6.08 / 6.08	5.75 / 5.62	+7.0%
water cycle	6.51 / 6.50	6.53 / 6.52	0%
rivers	5.79 / 5.82	5.24 / 5.57	+7.4%

The earlier +15% estimate came from a low 5.12 baseline and is retired. The gain is workload-dependent for a structural reason: the kernel only accelerates experts computed on CPU, and the water-cycle prompt's routing happens to be served almost entirely from the GPU hot tier. This is the clearest evidence yet that the next big lever is making GPU experts a compute tier (Tensor Core grouped GEMM), which would also shrink every CPU-side gain — and, by making multi-position verify batches cheap, would re-open the MTP speculation door (69-79% acceptance is already banked above).

Verdict: the kernel is numerically superior, quality-neutral, and worth +4-7% on CPU-heavy routings at zero cost. Recommended for merge; the only open item is the policy call that outputs are not bit-identical to the old (worse) accumulation order. Logs: /data/test/colibri-six-gpu-ab-avx512q-*-20260713.log, /data/test/score-acc{0,1}.out, harness /data/test/i4_numeric_test.c.

Why six 5090s did not match two RTX PRO 6000s

The reference vLLM-Moet result reports roughly 28-32 tok/s on two RTX PRO 6000 Blackwell cards. Each PRO 6000 has 96 GB of VRAM, while each RTX 5090 has 32 GB. Both machines may total 192 GB, but expert residency is constrained per rank and runtime state is replicated. Six smaller memory islands are not equivalent to two large ones.

The 5090 has strong compute and is cost-effective for concurrency, but this workload is dominated by resident expert capacity and miss recovery. Changing TP/PP layout cannot repair a policy that replays an entire step after any expert miss.

Conclusions for colibri

The core idea remains valid: VRAM, RAM, and NVMe are storage tiers with different bandwidth and latency, so a model larger than VRAM can run. The full-resident experiments strengthened it: on this machine the whole int4 model fits in VRAM+RAM, and once disk left the decode path colibri at 6.28-6.84 tok/s beat every vLLM-Moet layout measured here (best 2.6 tok/s), because colibri never replays a whole step on an expert miss.

The remaining bottleneck is no longer storage or scheduling — it is the int4 expert matmul on CPU (56% of step time, 84.9% of CPU samples in matmul_qt), with attention (25%) next in line. Priorities:

1. Qualify the AVX-512 int4 kernel (numeric-error tests, perplexity evaluation, fixed greedy repeat runs) and take its ~15% if it passes.
2. Add finer counters to CPU expert execution (experts per layer, rows per expert, per-matrix time) to guide kernel work.
3. Keep all A/B tests on fixed TEMP=0 prompts and fixed token counts; never mix sampled runs into comparisons.
4. Once the expert path is stably above about 7 tok/s, optimize attention projection/RoPE and score-softmax-value.

Twenty to thirty tokens per second has still not been demonstrated on this machine; 6.84 tok/s (256-token greedy) is the current ceiling.

Last known lab state

The fastest configuration overall is colibri full-resident in /data/test/colibri-full-resident (6.28-6.84 tok/s; stop the glm52-moet-real service before benchmarking it and restore the service afterwards). The fastest vLLM-Moet configuration was the TP4 service on port 8000, using GPUs 0-3 and the 14 GiB-per-card expert cache; GPUs 4-5 were idle. Preserve both pack directories until the PP experiment no longer needs to be reproduced. Raw A/B logs live under /data/test/colibri-six-gpu-ab-*.log. These paths describe the lab machine and are not repository dependencies.