danielkrause/colibri-strix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
76 Commits 1 Branch 0 Tags
92011def0746a2b53d7a625f320a89e29db882e5
Commit Graph

42 Commits

Author SHA1 Message Date
JustVugg 92011def07 README: generalize the reproducibility note (#100) — quantized-kernel rounding forks greedy across MTP/CUDA/batched, not just MTP; exact-mode recipe DRAFT=0 IDOT=0 COLI_CUDA=0 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 23:12:46 +02:00
JustVugg 8e526c68a9 README: lead with the int8-MTP mirror, make the int4-head 0%-acceptance trap unmissable (#8, #102) — the #1 support confusion 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 23:08:36 +02:00
JustVugg 2baac89fee README: corrected 9800X3D benchmark row (#101) — 10.5 GB/s VHDX still disk-bound, CUDA tier ~0% when AVX-512 CPU matches the 5090; CUDA value is CPU-dependent 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 22:49:29 +02:00
JustVugg f1fbbca352 README honesty fixes: MTP not byte-identical to greedy (shape-dependent kernels, #100); VHDX ceiling was this drive not the virtualization layer (#101, community measured 10.5 GB/s) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 22:08:13 +02:00
JustVugg e945b43842 README: Metal backend M5 Max benchmark — 1.83 tok/s, fastest datapoint yet (#72, #87) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 14:08:40 +02:00
JustVugg 928fd67cc2 README: correct M5 Max disk number (14.2 GB/s was cache-influenced ~4 GB/s real) + iobench cold-read caveat, macOS F_NOCACHE limitation (#86) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 14:04:11 +02:00
Sidd 2416bc9079 Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85) 
* feat: standardize runtime output in English

* test: cover English CLI output
 2026-07-12 13:38:40 +02:00
Fabio Rovai cec7d6b648 Grammar-forced speculative drafts: GBNF grammar as a third draft source, guaranteed-accepted forced spans, lossless + opt-in (#48, #70) 
New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA
walker) wired into spec_decode as a third draft source ("metodo F"), tried
before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the
forced span is tokenized and injected as drafts; the existing batch-union
verification confirms them, so a wrong or out-of-sync grammar can never
change the output. Lazy arming skips preambles; adaptive guard (same
pattern as MTP) disables the source below 50% acceptance; grammar-accepted
tokens no longer pollute the MTP acceptance counter.

GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and
with the int4 MTP head from #8); GRAMMAR_DRAFT=n caps the span (default 24).

Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON
classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok),
100% acceptance (48/48), output byte-identical to baseline.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-12 01:35:39 +02:00
ZacharyZcR 8f5f3e3a2b coli doctor: read-only setup/health diagnostics (path, config, shards, disk, RAM budget, placement) (#33) 
* Add read-only coli doctor diagnostics

* Fix doctor JSON output assertion
 2026-07-11 19:39:13 +02:00
nalepy 89d95fc73b Windows 11 native port, phase 1: MinGW-w64 static build, compat shims, setup + docs (#40) 
* fix: Windows port audit fixes — _FILE_OFFSET_BITS guard, O_BINARY st.h, getrusage peak, oracle diagnostic, setup.sh wmic

Audit remediation (all MEDIUM issues fixed):
- compat.h: compile-time #error if _FILE_OFFSET_BITS < 64 on _WIN32
- compat.h: COMPAT_O_RDONLY macro (O_RDONLY|O_BINARY on Windows, belt-and-braces)
- st.h: use COMPAT_O_RDONLY in both open() call sites (plan §1 row 1)
- compat.h: getrusage shim now uses PeakWorkingSetSize (ru_maxrss = peak, not current)
- glm.c: oracle mismatch diagnostic — prints position/expected/got on TF failures
- setup.sh: replace deprecated wmic with /proc/meminfo (MSYS2 provides it)
- .gitignore: *.exe, glm_tiny/, olmoe_hf/, olmoe_i4/

LOW issues addressed:
- _FILE_OFFSET_BITS guard prevents silent 32-bit off_t wrap at >4GB offsets
- coli Windows venv path (Scripts/python.exe) fixed earlier
- posix_fadvise do{}while(0) kept intentionally — no caller checks return code

Verification: oracle 32/32, 27/27 Python tests, rename EEXIST, >4GB pread offset.

* docs: add Windows 11 native port section to README

- Toolchain: MinGW-w64 (winlibs or MSYS2), GCC 16.1 tested
- Build instructions: make glm.exe, tiny oracle verification
- Runtime: SNAP=..., coli chat, coli serve all work
- Status: Phase 1 complete (compiles, correct, static-linked)
- Update platform requirements to include Windows 11 natively
 2026-07-11 12:59:49 +02:00
JustVugg a5fc89e88f README: Strix Halo warm numbers (#39) — 0.16 -> 0.40 tok/s in five runs, learned pin 47.6 GB, fastest non-Apple datapoint 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-11 01:25:59 +02:00
JustVugg ae8975a009 README: Strix Halo + Optane 905p datapoint (#39) — first Ryzen AI Max+, Optane latency invisible to 19MB-chunk reads 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-11 00:13:56 +02:00
JustVugg c809bc8260 README: 9950X disk-swap datapoint (#31) — same machine, P3 vs 9100 PRO: x5.8 disk = x2.9 tokens, profile flips 66% disk -> 57% matmul 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-11 00:01:56 +02:00
ZacharyZcR 9e6e1ac327 Isolated sequence KV contexts: KVState extraction, KV_SLOTS serve contexts with per-slot persistence, budget-aware pool accounting (#29) 
* Add isolated sequence KV contexts

* Log projected multi-slot KV footprint
 2026-07-10 16:19:43 +02:00
ZacharyZcR 76e858a80f Bounded FIFO inference scheduling: admission queue, OpenAI-shaped 429/503, queued-disconnect detection (#28) 2026-07-10 12:52:07 +02:00
ZacharyZcR 3cbc52ab00 coli plan: read-only Disk/RAM/VRAM placement planner (mirrors engine budget math, versioned --json, --auto-tier) (#27) 
* Add tiered resource planner

* Apply resource plans to runner configuration

---------

Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
 2026-07-10 12:51:23 +02:00
ZacharyZcR 1453dab7ae REPIN follows live session heat (decaying heat map; .coli_usage stays the persistent signal) + disk→RAM→VRAM promotion (#26) 2026-07-10 12:48:53 +02:00
ZacharyZcR 3e4d08b6bf OpenAI-compatible HTTP API: stdlib-only gateway over SERVE with KV prefix reuse across stateless requests (#21) 
* Add OpenAI-compatible HTTP API

* Support browser API clients

* Handle missing KV cache during rewind
 2026-07-10 11:04:56 +02:00
JustVugg 1a909f3833 README: catch up with the last two days — DSA is done (was 'in progress'), KV persistence, PILOT, web/ UI section + repo layout, honest line count 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 10:12:40 +02:00
ZacharyZcR 13e8f09ffc Organize project tools and local workflows: c/tools, c/scripts, c/tests, root Makefile (flat C core untouched) (#22) 2026-07-10 10:08:39 +02:00
JustVugg 8f99e12b5e KV-to-disk: conversations reopen WARM across engine restarts (.coli_kv, KVSAVE=0 opts out) 
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.

Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 10:00:19 +02:00
JustVugg 99111993a4 README: Framework 13 warm numbers (0.37 tok/s, hit 66%, MTP 52% — #12), int8-MTP mirror clone link (#2), Epyc row column fix 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 07:46:18 +02:00
JustVugg 077894210e LOOKA: routing predictability counters; PILOT: router-lookahead disk prefetch (async I/O thread); cap auto-raise to RAM budget (#12) 
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.

cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).

DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 07:44:26 +02:00
ZacharyZcR 57706a0200 Tiered CUDA acceleration for routed experts (opt-in, CPU default untouched) + REPLAY fixture harness (#16) 
* feat: add experimental CUDA backend for resident tensors

* feat: promote pinned experts to a bounded VRAM tier

* feat: preload the GPU expert tier at startup

* fix: harden CUDA backend failure handling

* feat: add deterministic multi-GPU tensor placement

* test: add deterministic CUDA benchmark fixture

* perf: make routed experts the default CUDA path
 2026-07-10 07:41:09 +02:00
adapt-L 4ea9ddc0f0 README: community benchmark — Epyc 9654 ES, PCIe3 NVMe (0.31 tok/s, hit 35%) (#17) 
I think I am just disk bandwidth limited, CPU's are at like 65% average. Might consider upgrading to 4x PCIe5 x4 ssds...
 2026-07-10 07:41:06 +02:00
JustVugg ed3916b0bb README: accurate SSD note (reads don't wear the drive; swap + thermals are the real concerns) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-10 01:39:50 +02:00
JustVugg c035f41778 MTP head is int8 by default (int4 head: 0-4% acceptance, unusable — measured in #8); README: honest MTP numbers + M5 Max community datapoint (#4, #5) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 20:34:08 +02:00
DatPat 996ae0e3cd README: add measured community benchmark (WSL2, 24 GB RAM — issue #2) (#3) 
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-08 10:20:37 +02:00
Vincenzo 4319dfdf91 Add model download huggingface section to README 
Added download instructions for the GLM-5.2 int4 model on Hugging Face.
 2026-07-08 08:42:44 +02:00
JustVugg 492c3b6f63 Revert "coli: multi-model registry + picker (coli models / chat menu); README: MoE-only model roadmap (gpt-oss-120b next)" 
This reverts commit 289befb0fd.
 2026-07-07 13:03:07 +02:00
JustVugg 289befb0fd coli: multi-model registry + picker (coli models / chat menu); README: MoE-only model roadmap (gpt-oss-120b next) 
Each architecture maps to its own engine binary (glm today; gptoss, qwenmoe
reserved). Registry in c/models.json (local, gitignored); chat shows a picker
when more than one model is installed. Dense models stay llama.cpp territory
- documented honestly in the README.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-07 12:59:56 +02:00
JustVugg b71d3884f6 coli bench: self-contained (venv python + auto-fetch datasets); README calls for a quality run on faster hardware 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-07 07:29:52 +02:00
JustVugg 8d7fbef39c README: today's engine work (integer kernels, MLA absorption, readahead, sampling, DSA status) 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-06 20:24:18 +02:00
JustVugg 34fb900762 README: honest matmul-bound predictions (GFLOP math), updated sampling defaults 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-06 19:40:27 +02:00
JustVugg b2855c0da1 coli convert: one command does model + MTP head; README documents the full zero-to-chat path 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-06 16:59:27 +02:00
Vincenzo 8e8d5fb02b Update README.md 2026-07-05 23:22:23 +02:00
Vincenzo ea27cd77fb Update README.md 2026-07-05 23:18:00 +02:00
Vincenzo 1e33e9b17d Update README.md 2026-07-05 23:16:58 +02:00
JustVugg 31d91b2c5b README: TUI pixel-art hummingbird as SVG logo banner 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-05 21:15:09 +02:00
JustVugg 20008742bf README: try-it-on-better-hardware guide + predictions + support section; drop stage-0 research scripts from repo 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-05 21:02:46 +02:00
JustVugg 1ae22a6135 colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts 
Engine (c/glm.c): MLA attention with compressed KV, sigmoid noaux_tc router,
int8/int4/int2 quant kernels (AVX2), per-layer LRU expert cache + pinned
hot-store, batch-union MoE, native MTP speculative decoding (lossless),
multi-stop + official chat template, RAM auto-budget from MemAvailable.
Tokenizer: byte-level BPE in C. Tooling: coli CLI, disk-safe FP8→int4
converter, tiny-random oracle validation (TF 32/32, greedy 20/20).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-05 20:57:25 +02:00
Vincenzo a6deef8e44 Initial commit 2026-07-01 14:27:49 +02:00