coli bench: self-contained (venv python + auto-fetch datasets); README calls for a quality run on faster hardware

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 07:29:52 +02:00
parent 193d2ce92d
commit b71d3884f6
2 changed files with 26 additions and 3 deletions
@@ -114,6 +114,19 @@ PIN=stats.txt PIN_GB=20 ./coli chat        # scale PIN_GB to your free RAM

 These are estimates, not measurements — if you run colibrì on serious hardware, **please open an issue with your numbers**: real datapoints from better machines are exactly what this project needs next.

+## Quality benchmark — help wanted
+
+We have never measured how much the int4 quantization costs in accuracy — the harness is built and wired, but scoring is one forward per answer option, and on the dev box's ~1 GB/s disk a full run takes the better part of a day. **This is the single most valuable thing a faster machine can contribute.** The code is here and ready; one command runs it end to end (it auto-downloads the datasets on first use):
+
+```bash
+cd c
+./coli bench                                   # hellaswag, arc_challenge, mmlu — 40 questions each
+./coli bench hellaswag --limit 200             # one task, more questions
+./coli bench mmlu arc_challenge --ram 100      # pick tasks, set a RAM budget
+```
+
+It prints per-task accuracy (log-likelihood scoring, EleutherAI-harness style). Published full-precision GLM-5.2 scores on these tasks sit around 85–95%; if our int4 container lands within a few points, the quantization is validated — if it doesn't, we know to invest in mixed / grouped-scale quantization. **If you have the hardware to run this, please open an issue with the numbers** — it's the measurement the project is missing.
+
 ## Supporting the project

 colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better test hardware translates *directly* into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can: