- Faithful to the official modeling (transformers glm_moe_dsa): q from the
q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
machinery forced on with keep=all keys reproduces dense attention exactly
(TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JustVugg
193d2ce92d