docs: GLM-5.2 6x RTX 5090 full-residency experiment — 6.84 tok/s, honest dead-ends log (#94)

Records the July 12-13 lab findings on one 6x RTX 5090 machine:
- vLLM-Moet comparison and why per-rank expert residency dominates
- colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the
  tested-and-rejected directions documented
- MTP speculation: broken int4 head identified (issue #8), int8 head
  reaches 69-79% acceptance but MoE verify batches scale expert cost
  with S, so speculation loses at every depth; revisit after GPU
  grouped GEMM
- AVX-512 int4 kernel qualification: numerically better than the old
  order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing

README links the record from the honest-numbers section.
This commit is contained in:
ZacharyZcR
2026-07-13 04:06:51 +08:00
committed by GitHub
parent f38e74efe7
commit 3fb5a00106
2 changed files with 304 additions and 0 deletions
+2
View File
@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
| metric | value |
|---|---|
| model on disk (int4 container) | ~370 GB |