docs: GLM-5.2 6x RTX 5090 full-residency experiment — 6.84 tok/s, honest dead-ends log (#94)
Records the July 12-13 lab findings on one 6x RTX 5090 machine: - vLLM-Moet comparison and why per-rank expert residency dominates - colibri full-resident placement: 2.30 -> 6.28-6.84 tok/s, with the tested-and-rejected directions documented - MTP speculation: broken int4 head identified (issue #8), int8 head reaches 69-79% acceptance but MoE verify batches scale expert cost with S, so speculation loses at every depth; revisit after GPU grouped GEMM - AVX-512 int4 kernel qualification: numerically better than the old order, quality-neutral (ppl delta 0.24%), +4-7% on CPU-heavy routing README links the record from the honest-numbers section.
This commit is contained in:
@@ -43,6 +43,8 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
|
||||
|
||||
## Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)
|
||||
|
||||
Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-2026-07-12.md) — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
|
||||
|
||||
| metric | value |
|
||||
|---|---|
|
||||
| model on disk (int4 container) | ~370 GB |
|
||||
|
||||
Reference in New Issue
Block a user