Rebased onto current dev, split into 3 logical parts (all validated):
1. CPU portability (serve-mode _O_BINARY pipe fix — stock main hangs on MinGW without it; RAM detection cap 0->9/layer; POSIX guards for select/mmap/madvise; warmup script).
2. AVX-VNNI 128-bit int8/int4 dot kernel (Alder Lake+/Meteor Lake+), bit-identical to AVX2 (author-verified on Meteor Lake; compiles out to AVX2 elsewhere) + _mm256_extracti128_si256 typo fix that blocked -march=native.
3. CUDA DLL via LoadLibrary, gated behind CUDA_DLL=1 (host never links cudart; silent CPU fallback if absent; author-verified on RTX 5070 Ti).
Validated here: make check 59/59, oracle 32/32 TF, Windows cross-compile clean + glm.exe loads+runs via WSL interop. Fixes the #123 Windows build failure.
env_for() mapped --gpu/--vram only inside the --auto-tier branch, so the standalone form started a CPU-only engine with no warning (#121 — nearly published as a GPU benchmark). Now: --gpu none disables CUDA; --gpu list/auto sets COLI_CUDA/COLI_GPUS; --vram sets CUDA_EXPERT_GB; and --gpu/--vram on a CPU-only binary exit with a clear 'make glm CUDA=1' message instead of falling back silently. Tested on a CPU-only build: fail-fast fires for --gpu/--vram, --gpu none and default still start the CPU engine (no regression).
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fenced code blocks become bordered boxes with the language label, **bold**
renders as real bold, `inline code` colored, # headers, - bullets. Works
char-by-char on the live stream (markers split across chunks are held back),
inline state resets per line, and orphan ``` fences right after a close
(a known int4 glitch) are swallowed. COLI_RAW=1 restores raw output.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Faithful to the official modeling (transformers glm_moe_dsa): q from the
q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
machinery forced on with keep=all keys reproduces dense attention exactly
(TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Learning cache: expert usage persists in <SNAP>/.coli_usage across sessions
(atomic save every turn); at startup the hottest experts are auto-pinned in
RAM with half the expert budget (AUTOPIN=0 disables). The engine gets faster
the more you use it.
- Sampling: temperature + nucleus (official 1.0/0.95 defaults in chat; TEMP=0
= greedy). MTP/n-gram speculation stays lossless via rejection sampling
(accept draft w.p. p(draft); on reject resample with draft banned).
- coli: --temp flag.
- Converter: --indexer mode extracts DSA lightning-indexer weights
(resumable; needed for future sparse attention beyond 2048 ctx).
- pin_load/stats include the MTP row; usage histogram covers layer 78.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cody
woolcoxm