* openai_server: parse GLM-5.2 tool calls into OpenAI tool_calls
Fills the 'tools not supported yet' stub. Upstream declared nothing and
rejected tools/functions; this makes them real end to end:
render_chat: emit a tools-declaration <|system|> block; replay assistant
tool_calls as <tool_call>NAME<arg_key>K</arg_key><arg_value>V
</arg_value></tool_call>; render tool-result messages as
<|observation|><tool_response>...</tool_response> (one
observation per consecutive tool run).
parse_tool_calls: turn the model's <tool_call> markers back into OpenAI
tool_calls; strip <think> from surfaced content.
generation: non-stream returns message.tool_calls + finish_reason
'tool_calls'; stream suppresses the markers from content
deltas (marker-length hold-back so a split <tool_call> is
caught) and emits tool_calls deltas after generation.
Default-safe: with no tools in the request, render and streaming take the
exact upstream path (byte-identical); tool handling is gated on tools present.
Marker format is authoritative (GLM-5.2 chat_template.jinja).
Optional de-mangler (COLI_TOOL_SALVAGE=1, default OFF) recovers malformed
calls from heavily-quantized models by mapping a lone payload onto the tool's
primary parameter; never rewrites well-formed output, never on by default.
A [api] tool-calls telemetry line reports strict vs de-mangled.
Pure stdlib; no new deps.
* openai_server: COLI_THINK global thinking default (opt-in)
COLI_THINK=1 makes thinking the default when a request sends neither
reasoning_effort nor enable_thinking (a launch-time global switch equivalent
to the old server's --think, useful because reasoningEffort propagation from
openai-compatible front-ends is unreliable). An explicit client value always
wins. Default off => exact OpenAI-standard behavior, so this is inert unless
enabled.
* openai_server: keepalive during long prefill (SSE reasoning pings)
engine.generate() blocks silently through the cold prefill (minutes for a
large prompt), and OpenCode/undici drop the socket after their idle timeout
-> the stream is 'terminated' before the first token. Upstream's SSE path
emits nothing until generation produces text, so it has no heartbeat.
A background pump emits a reasoning_content '.' delta whenever no event has
been written for KA_GAP (10s): the channel that reliably resets the client
timer and lands in the thinking panel, so answer content stays clean. All
wfile writes share a lock so the pump and event() never interleave; a
last-write timestamp gates the pump so it's silent while real tokens flow
(decode). The pump stops as soon as generation returns.
Inert for fast responses (nothing sent if events flow within 10s). No new
deps.
* openai_server: COLI_DEBUG echoes decoded tokens to stderr
Upstream sends decoded tokens only to the client (stdout->SSE), so the
console shows no generation text -- painful when the client has disconnected
or for local debugging. COLI_DEBUG=1 tees each decoded chunk to stderr at the
engine-callback boundary (both the plain and tool-call streaming paths).
Default off; no effect unless enabled.
* openai_server: use GLM-5.2 authoritative tool-declaration block
The tools were declared with a hand-written preamble; GLM-5.2 was trained on
the '# Tools' + <tools></tools> XML structure from chat_template.jinja. With
the wrong preamble the model doesn't recognize the tools as native and
hallucinates other frameworks' syntax (observed: repetitive 'end_action' and
key+value concatenated into one arg). Switch render_chat to the byte-exact
trained format so the model emits well-formed <tool_call> blocks.
PowerPC GCC uses -mcpu instead of -march, so the Linux branch failed
with unrecognized option -march=native on ppc64le. Detect ppc64le and
ppc64 via uname -m and use -mcpu=$(ARCH) there. The x86-64 path is
unchanged.
Validated on an IBM POWER8 S824 (Ubuntu 20.04, gcc 9.4): make test-c
passes, teacher forcing 32/32 positions and greedy 20/20 tokens against
the transformers oracle, engine reports the scalar idot fallback.
Signed-off-by: Scott <scottbphone12@gmail.com>
Co-authored-by: Scott <scottbphone12@gmail.com>
For streamed (non-resident) experts the MoE forward is disk-bound: each
64-expert block first blocks on a parallel pread of every miss, then runs
the matmuls serially. Those two phases don't overlap, so the compute
cores idle while the block's weights are still coming off NVMe.
PIPE=1 hands the misses' expert_load() to a small persistent pool of I/O
worker pthreads and lets the main thread start matmul immediately. The
main thread walks the block's experts in routing order and waits on a
per-slot ready flag only for the expert it needs right now, so loads of
later experts in the block hide behind matmul of earlier ones. All
matmul_qt stays on the main thread (it parallelises internally via
OpenMP and gates GPU dispatch on !omp_in_parallel()), so the I/O pool
never competes with the compute team for the matmul itself.
Cross-generation correctness: generation-tagged lock-free cursor
--------------------------------------------------------------------
The batch state (njobs/eids[]/layer/ready[]) is reused in place across
64-blocks, so a straggler or late-woken worker could touch the NEXT
batch's state. Two prior fixes (a drain barrier, then an _Atomic active
counter) each still left a cross-generation window. Both are now removed
and replaced with a single generation-tagged cursor:
_Atomic uint64_t cur = (gen << 8) | index; // gen main-only, index 0..njobs
- dispatch writes njobs/layer/eids[]/ready-reset with RELAXED stores,
then RELEASE-stores cur (bumping gen); that release publishes all
batch state to any worker whose ACQUIRE-load of cur sees the new gen.
- a worker grabs a job by CAS-advancing the index; it reads eids[i]/
layer only AFTER the winning CAS. The CAS comparand carries the
generation, so if a new batch was published the stale CAS fails and
the worker re-reads — it can never grab a wrong-generation job or
read torn state, no matter where it was preempted (wake gap,
post-cursor, anywhere).
- gen is bumped only by the main thread and is monotonic ⇒ no ABA.
- the per-expert pipe_wait(ready[q]) in the matmul loop (kept) makes
every grabbed job complete before the block ends, so no grab
outlives its generation. That is what makes the old `active`
counter and the end-of-block drain barrier unnecessary — both are
removed. ready[] is reset before the publishing release, so no stale
flag survives into the new generation.
- the mutex/condvar now exist ONLY to park and wake idle workers, not
for correctness.
This only reorders I/O, never the computation: greedy decode output is
byte-identical to the blocking path.
Default OFF (PIPE=0) so upstream behaviour is unchanged; PIPE=1 opts in
and PIPE_WORKERS=n sizes the pool (default 8). No Makefile change: pure C
(stdatomic.h + sched.h), builds with the default toolchain.
Claude-Session: https://claude.ai/code/session_01DS7oc65c5RdA9V99otRCwt
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The MoE forward does many tiny, back-to-back per-expert matmul regions.
Under libgomp's default passive wait policy the worker team is parked
between regions, and the thread re-wake latency dominates the actual
compute. Switching the team to an active wait policy (with a bounded spin
count so long NVMe expert-load stalls still yield instead of burning a
core) keeps the threads hot across those regions.
Measured on the Zen5 build: expert-matmul wall time 66.9s -> 20.9s
(~3.2x on that phase). Output is numerically unchanged — this only
affects thread scheduling, not the computation.
Mechanism: libgomp parses OMP_/GOMP_ vars in a constructor that runs
before main(), so setenv() from main() and continuing is too late
(verified: the already-initialised runtime ignores it). Instead we seed
the winning defaults on first entry with overwrite=0 (so any value the
user already set wins), set a COLI_OMP_TUNED sentinel, and execv() self
once so a fresh libgomp constructor reads them. The sentinel guards the
exec, so we re-exec at most once. The block is the first statement in
main() so argv is passed verbatim to execv().
Discoverability / observability (review follow-up):
- COLI_NO_OMP_TUNE=1 is a documented kill-switch that disables the
whole re-exec + tuning path in one shot (distinct from the internal
COLI_OMP_TUNED re-exec sentinel);
- a one-line stderr breadcrumb is printed before the re-exec
("[OMP] hot-thread tuning: re-exec once (COLI_NO_OMP_TUNE=1 to skip)")
so the self-re-exec is self-documenting;
- execv only returns on failure, so a perror() now follows it
("execv self-reexec failed, running untuned") — a container without
/proc or a deleted inode falls back visibly instead of silently
losing the ~3.2x.
Fully opt-out / overridable and lossless:
- explicit OMP_WAIT_POLICY / GOMP_SPINCOUNT / OMP_PROC_BIND / OMP_DYNAMIC
in the environment win (overwrite=0);
- pre-setting COLI_OMP_TUNED=1 skips the re-exec entirely;
- COLI_NO_OMP_TUNE=1 disables the tuning path entirely;
- guarded by __linux__ (no-op re-exec elsewhere; the setenv defaults
still apply for any later-initialising runtime).
Claude-Session: https://claude.ai/code/session_01DS7oc65c5RdA9V99otRCwt
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The default ./glm mode validates against ref_glm.json, which is generated
from glm_tiny (a random tiny model). Pointed at the 744B GLM-5.2 it feeds
the model tiny-vocab prompt tokens and compares against tiny references —
0/20 and a degenerate loop, guaranteed on EVERY platform (x86/ARM/Metal),
which reads as an engine bug (it isn't). Detect the mismatch (real vocab +
tiny-range oracle ids) and print the correct commands instead. REF_FORCE=1
overrides. The engine self-test (SNAP=./glm_tiny TF=1) is unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a Nix flake pinned to nixos-26.05. Builds the glm engine with
gcc+OpenMP (AVX2 portable), wraps the coli Python CLI with pinned deps
(torch, safetensors, huggingface-hub, numpy), and provides a dev shell.
New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA
walker) wired into spec_decode as a third draft source ("metodo F"), tried
before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the
forced span is tokenized and injected as drafts; the existing batch-union
verification confirms them, so a wrong or out-of-sync grammar can never
change the output. Lazy arming skips preambles; adaptive guard (same
pattern as MTP) disables the source below 50% acceptance; grammar-accepted
tokens no longer pollute the MTP acceptance counter.
GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and
with the int4 MTP head from #8); GRAMMAR_DRAFT=n caps the span (default 24).
Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON
classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok),
100% acceptance (48/48), output byte-identical to baseline.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: Windows port audit fixes — _FILE_OFFSET_BITS guard, O_BINARY st.h, getrusage peak, oracle diagnostic, setup.sh wmic
Audit remediation (all MEDIUM issues fixed):
- compat.h: compile-time #error if _FILE_OFFSET_BITS < 64 on _WIN32
- compat.h: COMPAT_O_RDONLY macro (O_RDONLY|O_BINARY on Windows, belt-and-braces)
- st.h: use COMPAT_O_RDONLY in both open() call sites (plan §1 row 1)
- compat.h: getrusage shim now uses PeakWorkingSetSize (ru_maxrss = peak, not current)
- glm.c: oracle mismatch diagnostic — prints position/expected/got on TF failures
- setup.sh: replace deprecated wmic with /proc/meminfo (MSYS2 provides it)
- .gitignore: *.exe, glm_tiny/, olmoe_hf/, olmoe_i4/
LOW issues addressed:
- _FILE_OFFSET_BITS guard prevents silent 32-bit off_t wrap at >4GB offsets
- coli Windows venv path (Scripts/python.exe) fixed earlier
- posix_fadvise do{}while(0) kept intentionally — no caller checks return code
Verification: oracle 32/32, 27/27 Python tests, rename EEXIST, >4GB pread offset.
* docs: add Windows 11 native port section to README
- Toolchain: MinGW-w64 (winlibs or MSYS2), GCC 16.1 tested
- Build instructions: make glm.exe, tiny oracle verification
- Runtime: SNAP=..., coli chat, coli serve all work
- Status: Phase 1 complete (compiles, correct, static-linked)
- Update platform requirements to include Windows 11 natively
* fix: iobench Windows correctness — aligned free, 64-bit totals, full-range offsets
Three bugs found running iobench.exe on Windows 11 (MinGW-w64):
1. Heap corruption (0xC0000374): free() on posix_memalign'd buffer.
On Windows posix_memalign maps to _aligned_malloc, whose memory must
be released with _aligned_free. Use compat_aligned_free (plain free
on POSIX, _aligned_free on Windows).
2. Negative GB totals: 'long tot' is 32-bit on Windows (LLP64), so any
benchmark reading more than 2 GB overflowed. Now int64_t.
3. Inflated speeds: RAND_MAX is 32767 on Windows, so rand()*4096 only
generated offsets in the first 134 MB of the file — which the page
cache holds entirely after one pass, reporting RAM speed instead of
disk speed (measured 6 GB/s fake vs 2.1 GB/s real on a laptop NVMe).
Combine two rand() calls into 30 bits so offsets span the whole file.
Portable: same code path on Linux/macOS, same seeded sequence shape.
Model files come from untrusted mirrors. One choke point in load_cfg bounds
every dimension (hostile config.json now dies with a clear message instead
of reaching allocation sites); falloc guards the n*sizeof(float) wrap.
No hot-path calloc churn — the report's patch was declined in review, the
valid kernel of the idea is taken here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serve mode persists the compressed MLA KV-cache incrementally after every
turn (~182 KB/token appended, header count written last = crash-safe) and
resumes it at startup: the model remembers the whole conversation and zero
re-prefill happens. :reset and context-full restarts truncate the file.
The MTP layer's KV row is not saved; kv_start=-1 re-arms its decode window.
Validated: split-session answer byte-identical to an uninterrupted session
(tiny oracle, TEMP=0), and on the real 744B model a restarted chat resumed
58 tokens in 0.0s and recalled a fact from the previous session while
prefilling only the new question.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on GLM-5.2 (48 tok, greedy): next-layer routing is 71.6% predictable
one full layer ahead (79.4% skipping attention only; 41.3% previous-token).
PILOT=1 issues next-layer expert WILLNEED from a dedicated I/O thread while
the current layer computes — inline fadvise BLOCKS ~0.5ms/call on a saturated
disk queue (+92s/48 tok, measured), hence the lock-free ring + worker.
Neutral-in-noise on this dev box (disk already ~80% duty); expected to pay on
balanced machines (#12: 43% disk / 46% matmul) — opt-in, default off.
cap_for_ram now RAISES the LRU cap up to the RAM budget (ceiling n_experts,
CAP_RAISE=0 opt-out): big-RAM machines were silently running with cap=8
(#12: 128GB box using 22GB of a 110GB budget; #13: 92GB box, same).
DRAFT=3 on cold cache measured locally: 1399s vs 880s baseline for the same
48 tokens (acceptance 16%, experts/token 1809 vs 800) — confirms #8; DRAFT
re-evaluation belongs to warm-cache serve sessions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- dot_i8i8/dot_i4i8: guarded __AVX512VNNI__ branches using vpdpbusd
(64 bytes/iter, s32 accumulation, no 16-bit intermediate). AVX-512 has
no vpsignb, so the sign trick becomes abs(w) + mask-negate on x.
- dot_i4i8: nibble unpack stays 256-bit; activation blocks are permuted
to match the per-lane value order instead of re-sorting weights.
- The int4 IDOT gate (was hardcoded S>=2, from AVX2 measurements) becomes
g_i4s with a VNNI-aware default of 1: single-row decode goes through the
integer path where it now pays. I4S=n overrides for A/B; I4S=2 restores
the old behavior exactly. AVX2/NEON/scalar builds are unchanged.
- Startup banner reports the active idot kernel (avx512-vnni/avx2/neon/scalar).
Measured on Xeon 6 (24C, AVX512-VNNI), DDR5-5600, ~19.8 GB/s NVMe RAID0,
gcc 13.3 -march=native: 256-tok greedy decode 1.72 -> 2.08 tok/s (+21%),
expert-matmul -24%; second prompt +20%. Integer-path outputs bit-identical
(associative s32 adds, same no-saturation bounds); S=1 int4 adopts the same
int8-activation tradeoff IDOT already makes for S>=2.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
First real datapoint for the 'Got a better machine?' section: disk iobench
plus stock and --topp 0.7 inference numbers, with the RAM-bound takeaway.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
JustVugg