DSA lightning indexer: full implementation with auto-detection

- Faithful to the official modeling (transformers glm_moe_dsa): q from the
  q_a latent via wq_b (32 heads x 128), k = LayerNorm(wk(h)) shared across
  heads, interleaved RoPE on the first 64 dims, ReLU(q.k/sqrt(128)) weighted
  by weights_proj(h)/sqrt(32), causal top-2048 per query.
- 'full' layers compute the selection (+ maintain the indexer k-cache from
  token 0); 'shared' layers reuse it (IndexShare, index_topk_freq=4).
- Selection restricts both attention paths (absorbed decode + prefill
  reconstruction). MTP row stays dense.
- Auto-detected like MTP: if out-idx-* weights are present for all full
  layers, DSA arms itself; DSA=0 disables; DSA_FORCE/DSA_TOPK for testing.
- Validated on the tiny oracle (which ships indexer weights): selection
  machinery forced on with keep=all keys reproduces dense attention exactly
  (TF 32/32, gen 20/20); sparse smoke runs clean; kill switch verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
JustVugg
2026-07-06 22:07:50 +02:00
parent dd8c907800
commit 193d2ce92d
2 changed files with 145 additions and 15 deletions
+1 -1
View File
@@ -205,7 +205,7 @@ def cmd_chat(a):
mload=re.search(r"caricato in ([0-9.]+)s \| densa residente: ([0-9.]+) MB", elog)
if mload: print(f" {C.grn}✓{C.r} pronto in {mload.group(1)}s {C.dim}· residente {float(mload.group(2))/1000:.1f} GB · RSS {st.get('rss','?')} GB{C.r}")
for l in elog.splitlines(): # una riga di stato per riga, senza path
if l.startswith(("[RAM_GB","[PIN]","[MTP]","[USAGE]")):
if l.startswith(("[RAM_GB","[PIN]","[MTP]","[USAGE]","[DSA]")):
l=re.sub(r" ?\(?/[^ )]+\)?","",l.strip()) # via i percorsi lunghi
l=re.sub(r" da$| in$","",l)
for chunk in textwrap.wrap(l, term_w()-4) or [l]: