The default ./glm mode validates against ref_glm.json, which is generated
from glm_tiny (a random tiny model). Pointed at the 744B GLM-5.2 it feeds
the model tiny-vocab prompt tokens and compares against tiny references —
0/20 and a degenerate loop, guaranteed on EVERY platform (x86/ARM/Metal),
which reads as an engine bug (it isn't). Detect the mismatch (real vocab +
tiny-range oracle ids) and print the correct commands instead. REF_FORCE=1
overrides. The engine self-test (SNAP=./glm_tiny TF=1) is unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a Nix flake pinned to nixos-26.05. Builds the glm engine with
gcc+OpenMP (AVX2 portable), wraps the coli Python CLI with pinned deps
(torch, safetensors, huggingface-hub, numpy), and provides a dev shell.
New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA
walker) wired into spec_decode as a third draft source ("metodo F"), tried
before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the
forced span is tokenized and injected as drafts; the existing batch-union
verification confirms them, so a wrong or out-of-sync grammar can never
change the output. Lazy arming skips preambles; adaptive guard (same
pattern as MTP) disables the source below 50% acceptance; grammar-accepted
tokens no longer pollute the MTP acceptance counter.
GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and
with the int4 MTP head from #8); GRAMMAR_DRAFT=n caps the span (default 24).
Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON
classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok),
100% acceptance (48/48), output byte-identical to baseline.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: Windows port audit fixes — _FILE_OFFSET_BITS guard, O_BINARY st.h, getrusage peak, oracle diagnostic, setup.sh wmic
Audit remediation (all MEDIUM issues fixed):
- compat.h: compile-time #error if _FILE_OFFSET_BITS < 64 on _WIN32
- compat.h: COMPAT_O_RDONLY macro (O_RDONLY|O_BINARY on Windows, belt-and-braces)
- st.h: use COMPAT_O_RDONLY in both open() call sites (plan §1 row 1)
- compat.h: getrusage shim now uses PeakWorkingSetSize (ru_maxrss = peak, not current)
- glm.c: oracle mismatch diagnostic — prints position/expected/got on TF failures
- setup.sh: replace deprecated wmic with /proc/meminfo (MSYS2 provides it)
- .gitignore: *.exe, glm_tiny/, olmoe_hf/, olmoe_i4/
LOW issues addressed:
- _FILE_OFFSET_BITS guard prevents silent 32-bit off_t wrap at >4GB offsets
- coli Windows venv path (Scripts/python.exe) fixed earlier
- posix_fadvise do{}while(0) kept intentionally — no caller checks return code
Verification: oracle 32/32, 27/27 Python tests, rename EEXIST, >4GB pread offset.
* docs: add Windows 11 native port section to README
- Toolchain: MinGW-w64 (winlibs or MSYS2), GCC 16.1 tested
- Build instructions: make glm.exe, tiny oracle verification
- Runtime: SNAP=..., coli chat, coli serve all work
- Status: Phase 1 complete (compiles, correct, static-linked)
- Update platform requirements to include Windows 11 natively
* fix: iobench Windows correctness — aligned free, 64-bit totals, full-range offsets
Three bugs found running iobench.exe on Windows 11 (MinGW-w64):
1. Heap corruption (0xC0000374): free() on posix_memalign'd buffer.
On Windows posix_memalign maps to _aligned_malloc, whose memory must
be released with _aligned_free. Use compat_aligned_free (plain free
on POSIX, _aligned_free on Windows).
2. Negative GB totals: 'long tot' is 32-bit on Windows (LLP64), so any
benchmark reading more than 2 GB overflowed. Now int64_t.
3. Inflated speeds: RAND_MAX is 32767 on Windows, so rand()*4096 only
generated offsets in the first 134 MB of the file — which the page
cache holds entirely after one pass, reporting RAM speed instead of
disk speed (measured 6 GB/s fake vs 2.1 GB/s real on a laptop NVMe).
Combine two rand() calls into 30 bits so offsets span the whole file.
Portable: same code path on Linux/macOS, same seeded sequence shape.
Model files come from untrusted mirrors. One choke point in load_cfg bounds
every dimension (hostile config.json now dies with a clear message instead
of reaching allocation sites); falloc guards the n*sizeof(float) wrap.
No hot-path calloc churn — the report's patch was declined in review, the
valid kernel of the idea is taken here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.
cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).
DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- dot_i8i8/dot_i4i8: guarded __AVX512VNNI__ branches using vpdpbusd
(64 bytes/iter, s32 accumulation, no 16-bit intermediate). AVX-512 has
no vpsignb, so the sign trick becomes abs(w) + mask-negate on x.
- dot_i4i8: nibble unpack stays 256-bit; activation blocks are permuted
to match the per-lane value order instead of re-sorting weights.
- The int4 IDOT gate (was hardcoded S>=2, from AVX2 measurements) becomes
g_i4s with a VNNI-aware default of 1: single-row decode goes through the
integer path where it now pays. I4S=n overrides for A/B; I4S=2 restores
the old behavior exactly. AVX2/NEON/scalar builds are unchanged.
- Startup banner reports the active idot kernel (avx512-vnni/avx2/neon/scalar).
Measured on Xeon 6 (24C, AVX512-VNNI), DDR5-5600, ~19.8 GB/s NVMe RAID0,
gcc 13.3 -march=native: 256-tok greedy decode 1.72 -> 2.08 tok/s (+21%),
expert-matmul -24%; second prompt +20%. Integer-path outputs bit-identical
(associative s32 adds, same no-saturation bounds); S=1 int4 adopts the same
int8-activation tradeoff IDOT already makes for S>=2.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fenced code blocks become bordered boxes with the language label, **bold**
renders as real bold, `inline code` colored, # headers, - bullets. Works
char-by-char on the live stream (markers split across chunks are held back),
inline state resets per line, and orphan ``` fences right after a close
(a known int4 glitch) are swallowed. COLI_RAW=1 restores raw output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Faithful to the official modeling (transformers glm_moe_dsa): q from the
q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
machinery forced on with keep=all keys reproduces dense attention exactly
(TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JustVugg