- dot_i8i8/dot_i4i8: guarded __AVX512VNNI__ branches using vpdpbusd
(64 bytes/iter, s32 accumulation, no 16-bit intermediate). AVX-512 has
no vpsignb, so the sign trick becomes abs(w) + mask-negate on x.
- dot_i4i8: nibble unpack stays 256-bit; activation blocks are permuted
to match the per-lane value order instead of re-sorting weights.
- The int4 IDOT gate (was hardcoded S>=2, from AVX2 measurements) becomes
g_i4s with a VNNI-aware default of 1: single-row decode goes through the
integer path where it now pays. I4S=n overrides for A/B; I4S=2 restores
the old behavior exactly. AVX2/NEON/scalar builds are unchanged.
- Startup banner reports the active idot kernel (avx512-vnni/avx2/neon/scalar).
Measured on Xeon 6 (24C, AVX512-VNNI), DDR5-5600, ~19.8 GB/s NVMe RAID0,
gcc 13.3 -march=native: 256-tok greedy decode 1.72 -> 2.08 tok/s (+21%),
expert-matmul -24%; second prompt +20%. Integer-path outputs bit-identical
(associative s32 adds, same no-saturation bounds); S=1 int4 adopts the same
int8-activation tradeoff IDOT already makes for S>=2.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
- Faithful to the official modeling (transformers glm_moe_dsa): q from the
q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
machinery forced on with keep=all keys reproduces dense attention exactly
(TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Learning cache: expert usage persists in <SNAP>/.coli_usage across sessions
(atomic save every turn); at startup the hottest experts are auto-pinned in
RAM with half the expert budget (AUTOPIN=0 disables). The engine gets faster
the more you use it.
- Sampling: temperature + nucleus (official 1.0/0.95 defaults in chat; TEMP=0
= greedy). MTP/n-gram speculation stays lossless via rejection sampling
(accept draft w.p. p(draft); on reject resample with draft banned).
- coli: --temp flag.
- Converter: --indexer mode extracts DSA lightning-indexer weights
(resumable; needed for future sparse attention beyond 2048 ctx).
- pin_load/stats include the MTP row; usage histogram covers layer 78.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dog279