Merge main into dev: README honesty fixes (#100,#101) + #96 tool calling + #97 ppc64le (landed on main by mistake, syncing dev)

README.md

@@ -26,7 +26,7 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
 - **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
 - **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
 - **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
-- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless — *and stays lossless under sampling* via rejection sampling. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up — the adaptive guard and `DRAFT=0` are there for that.
+- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless *in exact arithmetic* — but **not byte-identical to non-speculative greedy in practice** ([#100](https://github.com/JustVugg/colibri/issues/100)): the batched verification forward (S>1) uses shape-dependent integer kernels that round differently from the S=1 path, so on a near-tie the greedy argmax can flip. Every emitted token is still the argmax of a valid forward — the continuation stays correct — it just isn't bit-for-bit the same stream; `DRAFT=0` (or the adaptive guard) gives byte-exact greedy. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up.
 - **Grammar-forced speculative drafts** (`GRAMMAR=file.gbnf`, [#48](https://github.com/JustVugg/colibri/issues/48)) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly **one** legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from [#8](https://github.com/JustVugg/colibri/issues/8). It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, `| ( ) ? * +`, comments); `GRAMMAR_DRAFT=n` caps the forced span per forward (default 24). Composes with `DRAFT`/MTP, which fill the free-text gaps between forced spans.
 - **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
 - **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
@@ -52,7 +52,7 @@ Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-
 | load time | ~30 s |
 | peak RSS during chat | ~20 GB (auto-capped) |
 | cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
-| disk ceiling (VHDX random) | ~1 GB/s → ~0.05–0.1 tok/s cold |
+| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
 | MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
 
 This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
@@ -333,7 +333,7 @@ thrashing. Persistent `.coli_usage` remains the long-term signal and is not deca
 
 ## Got a better machine? Try it — here's what to expect
 
-colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB/s). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
+colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
 
 **How to test it, in order:**
 

c/Makefile

@@ -33,6 +33,18 @@ CFLAGS  = -D_FILE_OFFSET_BITS=64 -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-
 LDFLAGS = -lm -fopenmp -static
 EXE     = .exe
 else
+UNAME_M := $(shell uname -m)
+ifneq (,$(filter ppc64le ppc64,$(UNAME_M)))
+# --- Linux PowerPC (POWER8/POWER9/POWER10) ---
+# PowerPC GCC uses -mcpu, not -march. ARCH=native works on gcc >= 4.7.
+# The AVX2/NEON kernels fall back to the portable scalar C path
+# (validated token-exact vs the transformers oracle on a POWER8 S824).
+CC      = gcc
+ARCH   ?= native
+CFLAGS  = -O3 -mcpu=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function
+LDFLAGS = -lm -fopenmp
+EXE     =
+else
 # --- Linux x86-64 (percorso originale, invariato) ---
 CC      = gcc
 # ARCH=native -> ottimizzato per QUESTA macchina (default, piu' veloce).
@@ -43,6 +55,7 @@ CFLAGS  = -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-m
 LDFLAGS = -lm -fopenmp
 EXE     =
 endif
+endif
 
 # CUDA=1 adds an opt-in backend for resident tensors. The default build remains
 # pure C and keeps the original zero-dependency runtime.

c/openai_server.py

@@ -151,7 +151,87 @@ def content_text(content, param):
     return "".join(parts)
 
 
-def render_chat(messages, enable_thinking=False, reasoning_effort=None):
+# ---- GLM-5.2 tool calling -----------------------------------------------------------------
+# The model expresses tool calls as ordinary text (from chat_template.jinja):
+#   <tool_call>{name}<arg_key>{k}</arg_key><arg_value>{v}</arg_value>...</tool_call>
+# and tool results come back as <|observation|><tool_response>{content}</tool_response>.
+# We render those markers into the prompt and parse them back into OpenAI `tool_calls`.
+import re
+
+BOX_START, BOX_END = "<tool_call>", "</tool_call>"
+TR_OPEN,  TR_CLOSE = "<tool_response>", "</tool_response>"
+THINK_OPEN, THINK_CLOSE = "<think>", "</think>"
+
+_BOX_RE  = re.compile(re.escape(BOX_START) + r"(.*?)" + re.escape(BOX_END), re.DOTALL)
+_ARG_RE  = re.compile(r"<arg_key>([^<]*)</arg_key><arg_value>(.*?)</arg_value>", re.DOTALL)
+_NAME_RE = re.compile(r"\s*([A-Za-z0-9_.\-]+)")
+_TAG_RE  = re.compile(r"</?arg_key>|</?arg_value>")
+
+# De-mangler: opt-in recovery for heavily-quantized models that drop the
+# <arg_key>K</arg_key><arg_value> structure. Default OFF (never rewrites well-formed output).
+_SALVAGE = os.environ.get("COLI_TOOL_SALVAGE", "0") == "1"
+
+
+def _tool_param_order(tools):
+    """name -> ordered param names (required first) from the request schema, for de-mangling."""
+    out = {}
+    for tool in (tools or []):
+        fn = tool.get("function", tool) if isinstance(tool, dict) else {}
+        name = fn.get("name")
+        if not name:
+            continue
+        params = ((fn.get("parameters") or {}).get("properties") or {})
+        required = list((fn.get("parameters") or {}).get("required") or [])
+        out[name] = required + [p for p in params if p not in required]
+    return out
+
+
+def parse_tool_calls(reply, tools=None):
+    """Return (content, tool_calls). Strict GLM parse; optional de-mangler (COLI_TOOL_SALVAGE=1)
+    rescues malformed int4 output by mapping a lone payload onto the tool's primary parameter."""
+    param_order = _tool_param_order(tools)
+    calls, salvaged = [], []
+    for match in _BOX_RE.finditer(reply):
+        inner = match.group(1)
+        name_match = _NAME_RE.match(inner)
+        name = name_match.group(1) if name_match else inner.strip()
+        args = {}
+        for arg in _ARG_RE.finditer(inner):
+            key, value = arg.group(1), arg.group(2)
+            try:
+                value = json.loads(value)
+            except (json.JSONDecodeError, TypeError):
+                pass
+            args[key] = value
+        if not args and _SALVAGE:
+            rest = inner[name_match.end():] if name_match else ""
+            payload = _TAG_RE.sub("", rest).strip()
+            if payload.startswith("(") and payload.endswith(")"):
+                payload = payload[1:-1].strip()
+            if payload:
+                key = (param_order.get(name) or ["input"])[0]
+                try:
+                    payload = json.loads(payload)
+                except (json.JSONDecodeError, TypeError, ValueError):
+                    pass
+                args = {key: payload}
+                salvaged.append(name)
+        calls.append({"id": "call_" + uuid.uuid4().hex[:24], "type": "function",
+                      "function": {"name": name, "arguments": json.dumps(args, ensure_ascii=False)}})
+    text = _BOX_RE.sub("", reply)
+    if THINK_CLOSE in text:
+        text = text.split(THINK_CLOSE, 1)[1]
+    text = text.replace(THINK_OPEN, "").replace(THINK_CLOSE, "")
+    if calls:
+        dm = len(salvaged)
+        sys.stderr.write("[api] tool-calls: %d total, %d strict, %d de-mangled [%s]%s\n"
+                         % (len(calls), len(calls) - dm, dm, "CLEAN" if dm == 0 else "DE-MANGLED",
+                            (" -> " + ", ".join(salvaged)) if dm else ""))
+        sys.stderr.flush()
+    return text.strip(), calls
+
+
+def render_chat(messages, enable_thinking=False, reasoning_effort=None, tools=None):
     """Render the text-only subset of the official GLM-5.2 chat template."""
     if not isinstance(messages, list) or not messages:
         raise APIError(400, "`messages` must be a non-empty array.", "messages")
@@ -159,20 +239,57 @@ def render_chat(messages, enable_thinking=False, reasoning_effort=None):
     if enable_thinking:
         effort = "High" if reasoning_effort == "high" else "Max"
         prompt.append(f"<|system|>Reasoning Effort: {effort}")
+    if tools:
+        # AUTHORITATIVE GLM-5.2 tool-declaration block (byte-matches chat_template.jinja): the
+        # `# Tools` + <tools></tools> XML structure is what the model was trained on. A made-up
+        # preamble makes it hallucinate other frameworks' syntax (e.g. `end_action`).
+        prompt.append("<|system|>\n# Tools\n\nYou may call one or more functions to assist with the "
+                      "user query.\n\nYou are provided with function signatures within <tools></tools> "
+                      "XML tags:\n<tools>\n")
+        for tool in tools:
+            fn = tool.get("function", tool) if isinstance(tool, dict) else {}
+            clean = {k: v for k, v in fn.items() if k not in ("defer_loading", "strict")}
+            prompt.append(json.dumps(clean, ensure_ascii=False) + "\n")
+        prompt.append("</tools>\n\nFor each function call, output the function name and arguments "
+                      "within the following XML format:\n<tool_call>{function-name}"
+                      "<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value>"
+                      "<arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>")
+    prev_tool = False
     for index, message in enumerate(messages):
         if not isinstance(message, dict):
             raise APIError(400, "Each message must be an object.", f"messages.{index}")
         role = message.get("role")
-        text = content_text(message.get("content"), f"messages.{index}.content")
         if role in ("system", "developer"):
-            prompt.append(f"<|system|>{text}")
+            prompt.append(f"<|system|>{content_text(message.get('content'), f'messages.{index}.content')}")
         elif role == "user":
-            prompt.append(f"<|user|>{text}")
+            prompt.append(f"<|user|>{content_text(message.get('content'), f'messages.{index}.content')}")
         elif role == "assistant":
+            # content may be null when the message is purely tool_calls
+            raw = message.get("content")
+            text = content_text(raw, f"messages.{index}.content") if raw is not None else ""
             prompt.append(f"<|assistant|><think></think>{text.strip()}")
+            for tc in (message.get("tool_calls") or []):
+                fn = tc.get("function", tc) if isinstance(tc, dict) else {}
+                args = fn.get("arguments", "{}")
+                if isinstance(args, str):
+                    try:
+                        args = json.loads(args)
+                    except (json.JSONDecodeError, TypeError):
+                        args = {}
+                prompt.append(BOX_START + (fn.get("name") or ""))
+                for key, value in (args or {}).items():
+                    prompt.append(f"<arg_key>{key}</arg_key><arg_value>"
+                                  + (value if isinstance(value, str)
+                                     else json.dumps(value, ensure_ascii=False)) + "</arg_value>")
+                prompt.append(BOX_END)
+        elif role == "tool":
+            if not prev_tool:                       # one <|observation|> per consecutive tool run
+                prompt.append("<|observation|>")
+            prompt.append(TR_OPEN + content_text(message.get("content"), f"messages.{index}.content") + TR_CLOSE)
         else:
             raise APIError(400, f"Unsupported message role: {role!r}.",
                            f"messages.{index}.role", "unsupported_role")
+        prev_tool = (role == "tool")
     prompt.append("<|assistant|><think>" if enable_thinking else
                   "<|assistant|><think></think>")
     return "".join(prompt)
@@ -181,9 +298,7 @@ def render_chat(messages, enable_thinking=False, reasoning_effort=None):
 def generation_options(body, limit):
     if body.get("n", 1) != 1:
         raise APIError(400, "Colibri currently supports `n=1` only.", "n", "unsupported_value")
-    for name in ("tools", "functions"):
+    # `tools`/`functions` are handled by render_chat (declaration) + parse_tool_calls (output).
-        if body.get(name):
-            raise APIError(400, f"`{name}` is not supported yet.", name, "unsupported_parameter")
     if body.get("stop") is not None:
         raise APIError(400, "Custom stop sequences are not supported yet.", "stop", "unsupported_parameter")
     if body.get("logprobs"):
@@ -427,6 +542,7 @@ class APIHandler(BaseHTTPRequestHandler):
 
     def generation(self, body, prompt, request_id, chat):
         maximum, temperature, top_p = generation_options(body, self.server.max_tokens)
+        tools = (body.get("tools") or body.get("functions") or None) if chat else None
         cache_slot = body.get("cache_slot", 0)
         if isinstance(cache_slot, bool) or not isinstance(cache_slot, int) or not 0 <= cache_slot < self.server.kv_slots:
             raise APIError(400, f"`cache_slot` must be an integer between 0 and {self.server.kv_slots - 1}.",
@@ -450,10 +566,18 @@ class APIHandler(BaseHTTPRequestHandler):
                 stats = self.server.engine.generate(
                     prompt, maximum, temperature, top_p, output.append, cache_slot)
                 text = "".join(output)
-                finish = "length" if stats["length_limited"] else "stop"
+                length_finish = "length" if stats["length_limited"] else "stop"
+                if chat and tools:
+                    content, calls = parse_tool_calls(text, tools)
+                    message = {"role": "assistant", "content": content or None, "refusal": None}
+                    if calls:
+                        message["tool_calls"] = calls
+                    finish = "tool_calls" if calls else length_finish
+                    choice = {"index": 0, "message": message, "logprobs": None, "finish_reason": finish}
+                else:
                     choice = ({"index": 0, "message": {"role": "assistant", "content": text,
-                           "refusal": None}, "logprobs": None, "finish_reason": finish} if chat else
+                               "refusal": None}, "logprobs": None, "finish_reason": length_finish} if chat else
-                          {"index": 0, "text": text, "logprobs": None, "finish_reason": finish})
+                              {"index": 0, "text": text, "logprobs": None, "finish_reason": length_finish})
                 self.send_json(200, {"id": completion_id, "object": object_name, "created": created,
                     "model": self.server.model_id, "choices": [choice], "usage": self.usage(stats)},
                     request_id, queue_headers)
@@ -469,6 +593,18 @@ class APIHandler(BaseHTTPRequestHandler):
             self.send_cors_headers()
             self.end_headers()
             connected = True
+            # KEEPALIVE: engine.generate() blocks SILENTLY during the (minutes-long) cold
+            # prefill, and the client drops the socket after its idle timeout. A background pump
+            # emits a reasoning_content "." delta (the channel that reliably resets the client's
+            # timer and lands in the thinking panel, so answer content stays clean) whenever no
+            # event has been written for KA_GAP seconds. All wfile writes share ka_lock so the
+            # pump and event() never interleave; last_write gates the pump so it stays quiet
+            # while real tokens are flowing (e.g. during decode).
+            ka_lock = threading.Lock()
+            last_write = [time.time()]
+            ka_stop = threading.Event()
+            KA_GAP = 10.0
+            dbg_echo = os.environ.get("COLI_DEBUG", "0") == "1"   # tee decoded tokens to stderr
 
             def event(choices, usage_marker=False):
                 nonlocal connected
@@ -478,13 +614,24 @@ class APIHandler(BaseHTTPRequestHandler):
                               "model": self.server.model_id, "choices": choices}
                 if include_usage:
                     event_body["usage"] = None if not usage_marker else usage_marker
-                try:
                 data = json.dumps(event_body, ensure_ascii=False, separators=(",", ":"))
+                with ka_lock:
+                    try:
                         self.wfile.write(f"data: {data}\n\n".encode())
                         self.wfile.flush()
+                        last_write[0] = time.time()
                     except OSError:
                         connected = False
 
+            def _keepalive():
+                ping = [{"index": 0, "delta": ({"reasoning_content": "."} if chat else {"content": ""}),
+                         "logprobs": None, "finish_reason": None}]
+                while not ka_stop.wait(1.0):
+                    if not connected:
+                        return
+                    if time.time() - last_write[0] >= KA_GAP:
+                        event(ping)
+
             if chat:
                 event([{"index": 0, "delta": {"role": "assistant", "content": ""},
                         "logprobs": None, "finish_reason": None}])
@@ -495,9 +642,54 @@ class APIHandler(BaseHTTPRequestHandler):
                           {"index": 0, "text": text, "logprobs": None, "finish_reason": None})
                 event([choice])
 
+            ka_thread = threading.Thread(target=_keepalive, daemon=True)
+            ka_thread.start()
+            if chat and tools:
+                # Suppress tool-call markers from the streamed content and parse the authoritative
+                # calls from the FULL reply after generation. Hold back a marker-length tail so a
+                # <tool_call> split across engine chunks is still caught.
+                sp = {"buf": "", "tool": False}
+                hold = len(BOX_START) - 1
+                raw = []
+                def emit_tools(chunk):
+                    raw.append(chunk)
+                    if dbg_echo:
+                        sys.stderr.write(chunk); sys.stderr.flush()
+                    if sp["tool"]:
+                        return
+                    sp["buf"] += chunk
+                    cut = sp["buf"].find(BOX_START)
+                    if cut >= 0:
+                        if cut:
+                            emit(sp["buf"][:cut])
+                        sp["buf"] = ""
+                        sp["tool"] = True
+                        return
+                    flush = max(0, len(sp["buf"]) - hold)
+                    if flush:
+                        emit(sp["buf"][:flush])
+                        sp["buf"] = sp["buf"][flush:]
                 stats = self.server.engine.generate(
-                prompt, maximum, temperature, top_p, emit, cache_slot)
+                    prompt, maximum, temperature, top_p, emit_tools, cache_slot)
+                if not sp["tool"] and sp["buf"]:
+                    emit(sp["buf"])                     # no tool call happened: flush held tail
+                _content, calls = parse_tool_calls("".join(raw), tools)
+                for i, tc in enumerate(calls):
+                    event([{"index": 0, "delta": {"tool_calls": [{"index": i, "id": tc["id"],
+                             "type": "function", "function": {"name": tc["function"]["name"],
+                             "arguments": tc["function"]["arguments"]}}]},
+                            "logprobs": None, "finish_reason": None}])
+                finish = "tool_calls" if calls else ("length" if stats["length_limited"] else "stop")
+            else:
+                def emit_plain(chunk):
+                    if dbg_echo:
+                        sys.stderr.write(chunk); sys.stderr.flush()
+                    emit(chunk)
+                stats = self.server.engine.generate(
+                    prompt, maximum, temperature, top_p, emit_plain, cache_slot)
                 finish = "length" if stats["length_limited"] else "stop"
+            ka_stop.set()                          # generation done: stop the keepalive pump
+            ka_thread.join(timeout=2)
             final_choice = ({"index": 0, "delta": {}, "logprobs": None, "finish_reason": finish}
                             if chat else {"index": 0, "text": "", "logprobs": None,
                                           "finish_reason": finish})
@@ -535,10 +727,17 @@ class APIHandler(BaseHTTPRequestHandler):
         if reasoning_effort not in efforts:
             raise APIError(400, "`reasoning_effort` must be none, minimal, low, medium, high, or xhigh.",
                            "reasoning_effort")
+        # COLI_THINK=1 makes thinking the default when the client sends NEITHER reasoning_effort
+        # nor enable_thinking (a global switch, like the old server's --think). An explicit
+        # client value always wins. Default off => exact OpenAI-standard behavior.
+        if (reasoning_effort is None and "enable_thinking" not in body
+                and os.environ.get("COLI_THINK", "0") == "1"):
+            reasoning_effort = "high"
         enable_thinking = body.get("enable_thinking", reasoning_effort not in (None, "none"))
         if not isinstance(enable_thinking, bool):
             raise APIError(400, "`enable_thinking` must be a boolean.", "enable_thinking")
-        prompt = render_chat(body.get("messages"), enable_thinking, reasoning_effort)
+        tools = body.get("tools") or body.get("functions") or None
+        prompt = render_chat(body.get("messages"), enable_thinking, reasoning_effort, tools)
         self.generation(body, prompt, request_id, True)
 
     def completion(self, body, request_id):