Merge main into dev: README honesty fixes (#100,#101) + #96 tool calling + #97 ppc64le (landed on main by mistake, syncing dev)
This commit is contained in:
JustVugg
@@ -26,7 +26,7 @@ The engine is a single C file (`c/glm.c`, ~2,400 lines) plus small headers. No B
|
||||
- **Faithful GLM-5.2 (`glm_moe_dsa`) forward** — validated token-exact against a `transformers` oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
|
||||
- **MLA attention** (q/kv-LoRA, interleaved partial RoPE) with **compressed KV-cache**: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
|
||||
- **DeepSeek-V3-style sigmoid router** (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
|
||||
- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless — *and stays lossless under sampling* via rejection sampling. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up — the adaptive guard and `DRAFT=0` are there for that.
|
||||
- **Native MTP speculative decoding** — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. **The head must be int8** (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, **2.2–2.8 tokens/forward** (community-measured, [#8](https://github.com/JustVugg/colibri/issues/8)). Lossless *in exact arithmetic* — but **not byte-identical to non-speculative greedy in practice** ([#100](https://github.com/JustVugg/colibri/issues/100)): the batched verification forward (S>1) uses shape-dependent integer kernels that round differently from the S=1 path, so on a near-tie the greedy argmax can flip. Every emitted token is still the argmax of a valid forward — the continuation stays correct — it just isn't bit-for-bit the same stream; `DRAFT=0` (or the adaptive guard) gives byte-exact greedy. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a **cold** cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net *time* loss until the cache/pin warms up.
|
||||
- **Grammar-forced speculative drafts** (`GRAMMAR=file.gbnf`, [#48](https://github.com/JustVugg/colibri/issues/48)) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly **one** legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from [#8](https://github.com/JustVugg/colibri/issues/8). It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, `| ( ) ? * +`, comments); `GRAMMAR_DRAFT=n` caps the forced span per forward (default 24). Composes with `DRAFT`/MTP, which fill the free-text gaps between forced spans.
|
||||
- **True sampling** — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
|
||||
- **Integer-dot kernels** (Q8_0-style int8 activations, AVX2 `maddubs`): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
|
||||
@@ -52,7 +52,7 @@ Detailed GPU experiment: [GLM-5.2 on 6x RTX 5090](docs/experiments/glm52-6x5090-
|
||||
| load time | ~30 s |
|
||||
| peak RSS during chat | ~20 GB (auto-capped) |
|
||||
| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
|
||||
| disk ceiling (VHDX random) | ~1 GB/s → ~0.05–0.1 tok/s cold |
|
||||
| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
|
||||
| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured ([#8](https://github.com/JustVugg/colibri/issues/8)) |
|
||||
|
||||
This is not fast. It is a 744B frontier-class model **answering correctly on a machine that costs less than one H100 fan**. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
|
||||
@@ -333,7 +333,7 @@ thrashing. Persistent `.coli_usage` remains the long-term signal and is not deca
|
||||
|
||||
## Got a better machine? Try it — here's what to expect
|
||||
|
||||
colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB/s). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
|
||||
colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, an older DRAM-less NVMe behind a WSL2 VHDX that measured ~1 GB/s random on *this* drive — note WSL2 VHDX is not inherently slow: a community 5090 box measured 10.5 GB/s O_DIRECT through one, [#101](https://github.com/JustVugg/colibri/issues/101)). **Every one of those constraints is a knob your machine can turn up.** The engine needs: Linux (or WSL2), macOS, or **Windows 11 natively (MinGW-w64)**; gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4/NTFS — never a network/9p mount).
|
||||
|
||||
**How to test it, in order:**
|
||||
|
||||
|
||||
+13
@@ -33,6 +33,18 @@ CFLAGS = -D_FILE_OFFSET_BITS=64 -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-
|
||||
LDFLAGS = -lm -fopenmp -static
|
||||
EXE = .exe
|
||||
else
|
||||
UNAME_M := $(shell uname -m)
|
||||
ifneq (,$(filter ppc64le ppc64,$(UNAME_M)))
|
||||
# --- Linux PowerPC (POWER8/POWER9/POWER10) ---
|
||||
# PowerPC GCC uses -mcpu, not -march. ARCH=native works on gcc >= 4.7.
|
||||
# The AVX2/NEON kernels fall back to the portable scalar C path
|
||||
# (validated token-exact vs the transformers oracle on a POWER8 S824).
|
||||
CC = gcc
|
||||
ARCH ?= native
|
||||
CFLAGS = -O3 -mcpu=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-misleading-indentation -Wno-unused-function
|
||||
LDFLAGS = -lm -fopenmp
|
||||
EXE =
|
||||
else
|
||||
# --- Linux x86-64 (percorso originale, invariato) ---
|
||||
CC = gcc
|
||||
# ARCH=native -> ottimizzato per QUESTA macchina (default, piu' veloce).
|
||||
@@ -43,6 +55,7 @@ CFLAGS = -O3 -march=$(ARCH) -fopenmp -Wall -Wextra -Wno-unused-parameter -Wno-m
|
||||
LDFLAGS = -lm -fopenmp
|
||||
EXE =
|
||||
endif
|
||||
endif
|
||||
|
||||
# CUDA=1 adds an opt-in backend for resident tensors. The default build remains
|
||||
# pure C and keeps the original zero-dependency runtime.
|
||||
|
||||
+212
-13
@@ -151,7 +151,87 @@ def content_text(content, param):
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def render_chat(messages, enable_thinking=False, reasoning_effort=None):
|
||||
# ---- GLM-5.2 tool calling -----------------------------------------------------------------
|
||||
# The model expresses tool calls as ordinary text (from chat_template.jinja):
|
||||
# <tool_call>{name}<arg_key>{k}</arg_key><arg_value>{v}</arg_value>...</tool_call>
|
||||
# and tool results come back as <|observation|><tool_response>{content}</tool_response>.
|
||||
# We render those markers into the prompt and parse them back into OpenAI `tool_calls`.
|
||||
import re
|
||||
|
||||
BOX_START, BOX_END = "<tool_call>", "</tool_call>"
|
||||
TR_OPEN, TR_CLOSE = "<tool_response>", "</tool_response>"
|
||||
THINK_OPEN, THINK_CLOSE = "<think>", "</think>"
|
||||
|
||||
_BOX_RE = re.compile(re.escape(BOX_START) + r"(.*?)" + re.escape(BOX_END), re.DOTALL)
|
||||
_ARG_RE = re.compile(r"<arg_key>([^<]*)</arg_key><arg_value>(.*?)</arg_value>", re.DOTALL)
|
||||
_NAME_RE = re.compile(r"\s*([A-Za-z0-9_.\-]+)")
|
||||
_TAG_RE = re.compile(r"</?arg_key>|</?arg_value>")
|
||||
|
||||
# De-mangler: opt-in recovery for heavily-quantized models that drop the
|
||||
# <arg_key>K</arg_key><arg_value> structure. Default OFF (never rewrites well-formed output).
|
||||
_SALVAGE = os.environ.get("COLI_TOOL_SALVAGE", "0") == "1"
|
||||
|
||||
|
||||
def _tool_param_order(tools):
|
||||
"""name -> ordered param names (required first) from the request schema, for de-mangling."""
|
||||
out = {}
|
||||
for tool in (tools or []):
|
||||
fn = tool.get("function", tool) if isinstance(tool, dict) else {}
|
||||
name = fn.get("name")
|
||||
if not name:
|
||||
continue
|
||||
params = ((fn.get("parameters") or {}).get("properties") or {})
|
||||
required = list((fn.get("parameters") or {}).get("required") or [])
|
||||
out[name] = required + [p for p in params if p not in required]
|
||||
return out
|
||||
|
||||
|
||||
def parse_tool_calls(reply, tools=None):
|
||||
"""Return (content, tool_calls). Strict GLM parse; optional de-mangler (COLI_TOOL_SALVAGE=1)
|
||||
rescues malformed int4 output by mapping a lone payload onto the tool's primary parameter."""
|
||||
param_order = _tool_param_order(tools)
|
||||
calls, salvaged = [], []
|
||||
for match in _BOX_RE.finditer(reply):
|
||||
inner = match.group(1)
|
||||
name_match = _NAME_RE.match(inner)
|
||||
name = name_match.group(1) if name_match else inner.strip()
|
||||
args = {}
|
||||
for arg in _ARG_RE.finditer(inner):
|
||||
key, value = arg.group(1), arg.group(2)
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
args[key] = value
|
||||
if not args and _SALVAGE:
|
||||
rest = inner[name_match.end():] if name_match else ""
|
||||
payload = _TAG_RE.sub("", rest).strip()
|
||||
if payload.startswith("(") and payload.endswith(")"):
|
||||
payload = payload[1:-1].strip()
|
||||
if payload:
|
||||
key = (param_order.get(name) or ["input"])[0]
|
||||
try:
|
||||
payload = json.loads(payload)
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
pass
|
||||
args = {key: payload}
|
||||
salvaged.append(name)
|
||||
calls.append({"id": "call_" + uuid.uuid4().hex[:24], "type": "function",
|
||||
"function": {"name": name, "arguments": json.dumps(args, ensure_ascii=False)}})
|
||||
text = _BOX_RE.sub("", reply)
|
||||
if THINK_CLOSE in text:
|
||||
text = text.split(THINK_CLOSE, 1)[1]
|
||||
text = text.replace(THINK_OPEN, "").replace(THINK_CLOSE, "")
|
||||
if calls:
|
||||
dm = len(salvaged)
|
||||
sys.stderr.write("[api] tool-calls: %d total, %d strict, %d de-mangled [%s]%s\n"
|
||||
% (len(calls), len(calls) - dm, dm, "CLEAN" if dm == 0 else "DE-MANGLED",
|
||||
(" -> " + ", ".join(salvaged)) if dm else ""))
|
||||
sys.stderr.flush()
|
||||
return text.strip(), calls
|
||||
|
||||
|
||||
def render_chat(messages, enable_thinking=False, reasoning_effort=None, tools=None):
|
||||
"""Render the text-only subset of the official GLM-5.2 chat template."""
|
||||
if not isinstance(messages, list) or not messages:
|
||||
raise APIError(400, "`messages` must be a non-empty array.", "messages")
|
||||
@@ -159,20 +239,57 @@ def render_chat(messages, enable_thinking=False, reasoning_effort=None):
|
||||
if enable_thinking:
|
||||
effort = "High" if reasoning_effort == "high" else "Max"
|
||||
prompt.append(f"<|system|>Reasoning Effort: {effort}")
|
||||
if tools:
|
||||
# AUTHORITATIVE GLM-5.2 tool-declaration block (byte-matches chat_template.jinja): the
|
||||
# `# Tools` + <tools></tools> XML structure is what the model was trained on. A made-up
|
||||
# preamble makes it hallucinate other frameworks' syntax (e.g. `end_action`).
|
||||
prompt.append("<|system|>\n# Tools\n\nYou may call one or more functions to assist with the "
|
||||
"user query.\n\nYou are provided with function signatures within <tools></tools> "
|
||||
"XML tags:\n<tools>\n")
|
||||
for tool in tools:
|
||||
fn = tool.get("function", tool) if isinstance(tool, dict) else {}
|
||||
clean = {k: v for k, v in fn.items() if k not in ("defer_loading", "strict")}
|
||||
prompt.append(json.dumps(clean, ensure_ascii=False) + "\n")
|
||||
prompt.append("</tools>\n\nFor each function call, output the function name and arguments "
|
||||
"within the following XML format:\n<tool_call>{function-name}"
|
||||
"<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value>"
|
||||
"<arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>")
|
||||
prev_tool = False
|
||||
for index, message in enumerate(messages):
|
||||
if not isinstance(message, dict):
|
||||
raise APIError(400, "Each message must be an object.", f"messages.{index}")
|
||||
role = message.get("role")
|
||||
text = content_text(message.get("content"), f"messages.{index}.content")
|
||||
if role in ("system", "developer"):
|
||||
prompt.append(f"<|system|>{text}")
|
||||
prompt.append(f"<|system|>{content_text(message.get('content'), f'messages.{index}.content')}")
|
||||
elif role == "user":
|
||||
prompt.append(f"<|user|>{text}")
|
||||
prompt.append(f"<|user|>{content_text(message.get('content'), f'messages.{index}.content')}")
|
||||
elif role == "assistant":
|
||||
# content may be null when the message is purely tool_calls
|
||||
raw = message.get("content")
|
||||
text = content_text(raw, f"messages.{index}.content") if raw is not None else ""
|
||||
prompt.append(f"<|assistant|><think></think>{text.strip()}")
|
||||
for tc in (message.get("tool_calls") or []):
|
||||
fn = tc.get("function", tc) if isinstance(tc, dict) else {}
|
||||
args = fn.get("arguments", "{}")
|
||||
if isinstance(args, str):
|
||||
try:
|
||||
args = json.loads(args)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
args = {}
|
||||
prompt.append(BOX_START + (fn.get("name") or ""))
|
||||
for key, value in (args or {}).items():
|
||||
prompt.append(f"<arg_key>{key}</arg_key><arg_value>"
|
||||
+ (value if isinstance(value, str)
|
||||
else json.dumps(value, ensure_ascii=False)) + "</arg_value>")
|
||||
prompt.append(BOX_END)
|
||||
elif role == "tool":
|
||||
if not prev_tool: # one <|observation|> per consecutive tool run
|
||||
prompt.append("<|observation|>")
|
||||
prompt.append(TR_OPEN + content_text(message.get("content"), f"messages.{index}.content") + TR_CLOSE)
|
||||
else:
|
||||
raise APIError(400, f"Unsupported message role: {role!r}.",
|
||||
f"messages.{index}.role", "unsupported_role")
|
||||
prev_tool = (role == "tool")
|
||||
prompt.append("<|assistant|><think>" if enable_thinking else
|
||||
"<|assistant|><think></think>")
|
||||
return "".join(prompt)
|
||||
@@ -181,9 +298,7 @@ def render_chat(messages, enable_thinking=False, reasoning_effort=None):
|
||||
def generation_options(body, limit):
|
||||
if body.get("n", 1) != 1:
|
||||
raise APIError(400, "Colibri currently supports `n=1` only.", "n", "unsupported_value")
|
||||
for name in ("tools", "functions"):
|
||||
if body.get(name):
|
||||
raise APIError(400, f"`{name}` is not supported yet.", name, "unsupported_parameter")
|
||||
# `tools`/`functions` are handled by render_chat (declaration) + parse_tool_calls (output).
|
||||
if body.get("stop") is not None:
|
||||
raise APIError(400, "Custom stop sequences are not supported yet.", "stop", "unsupported_parameter")
|
||||
if body.get("logprobs"):
|
||||
@@ -427,6 +542,7 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
|
||||
def generation(self, body, prompt, request_id, chat):
|
||||
maximum, temperature, top_p = generation_options(body, self.server.max_tokens)
|
||||
tools = (body.get("tools") or body.get("functions") or None) if chat else None
|
||||
cache_slot = body.get("cache_slot", 0)
|
||||
if isinstance(cache_slot, bool) or not isinstance(cache_slot, int) or not 0 <= cache_slot < self.server.kv_slots:
|
||||
raise APIError(400, f"`cache_slot` must be an integer between 0 and {self.server.kv_slots - 1}.",
|
||||
@@ -450,10 +566,18 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
stats = self.server.engine.generate(
|
||||
prompt, maximum, temperature, top_p, output.append, cache_slot)
|
||||
text = "".join(output)
|
||||
finish = "length" if stats["length_limited"] else "stop"
|
||||
length_finish = "length" if stats["length_limited"] else "stop"
|
||||
if chat and tools:
|
||||
content, calls = parse_tool_calls(text, tools)
|
||||
message = {"role": "assistant", "content": content or None, "refusal": None}
|
||||
if calls:
|
||||
message["tool_calls"] = calls
|
||||
finish = "tool_calls" if calls else length_finish
|
||||
choice = {"index": 0, "message": message, "logprobs": None, "finish_reason": finish}
|
||||
else:
|
||||
choice = ({"index": 0, "message": {"role": "assistant", "content": text,
|
||||
"refusal": None}, "logprobs": None, "finish_reason": finish} if chat else
|
||||
{"index": 0, "text": text, "logprobs": None, "finish_reason": finish})
|
||||
"refusal": None}, "logprobs": None, "finish_reason": length_finish} if chat else
|
||||
{"index": 0, "text": text, "logprobs": None, "finish_reason": length_finish})
|
||||
self.send_json(200, {"id": completion_id, "object": object_name, "created": created,
|
||||
"model": self.server.model_id, "choices": [choice], "usage": self.usage(stats)},
|
||||
request_id, queue_headers)
|
||||
@@ -469,6 +593,18 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
self.send_cors_headers()
|
||||
self.end_headers()
|
||||
connected = True
|
||||
# KEEPALIVE: engine.generate() blocks SILENTLY during the (minutes-long) cold
|
||||
# prefill, and the client drops the socket after its idle timeout. A background pump
|
||||
# emits a reasoning_content "." delta (the channel that reliably resets the client's
|
||||
# timer and lands in the thinking panel, so answer content stays clean) whenever no
|
||||
# event has been written for KA_GAP seconds. All wfile writes share ka_lock so the
|
||||
# pump and event() never interleave; last_write gates the pump so it stays quiet
|
||||
# while real tokens are flowing (e.g. during decode).
|
||||
ka_lock = threading.Lock()
|
||||
last_write = [time.time()]
|
||||
ka_stop = threading.Event()
|
||||
KA_GAP = 10.0
|
||||
dbg_echo = os.environ.get("COLI_DEBUG", "0") == "1" # tee decoded tokens to stderr
|
||||
|
||||
def event(choices, usage_marker=False):
|
||||
nonlocal connected
|
||||
@@ -478,13 +614,24 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
"model": self.server.model_id, "choices": choices}
|
||||
if include_usage:
|
||||
event_body["usage"] = None if not usage_marker else usage_marker
|
||||
try:
|
||||
data = json.dumps(event_body, ensure_ascii=False, separators=(",", ":"))
|
||||
with ka_lock:
|
||||
try:
|
||||
self.wfile.write(f"data: {data}\n\n".encode())
|
||||
self.wfile.flush()
|
||||
last_write[0] = time.time()
|
||||
except OSError:
|
||||
connected = False
|
||||
|
||||
def _keepalive():
|
||||
ping = [{"index": 0, "delta": ({"reasoning_content": "."} if chat else {"content": ""}),
|
||||
"logprobs": None, "finish_reason": None}]
|
||||
while not ka_stop.wait(1.0):
|
||||
if not connected:
|
||||
return
|
||||
if time.time() - last_write[0] >= KA_GAP:
|
||||
event(ping)
|
||||
|
||||
if chat:
|
||||
event([{"index": 0, "delta": {"role": "assistant", "content": ""},
|
||||
"logprobs": None, "finish_reason": None}])
|
||||
@@ -495,9 +642,54 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
{"index": 0, "text": text, "logprobs": None, "finish_reason": None})
|
||||
event([choice])
|
||||
|
||||
ka_thread = threading.Thread(target=_keepalive, daemon=True)
|
||||
ka_thread.start()
|
||||
if chat and tools:
|
||||
# Suppress tool-call markers from the streamed content and parse the authoritative
|
||||
# calls from the FULL reply after generation. Hold back a marker-length tail so a
|
||||
# <tool_call> split across engine chunks is still caught.
|
||||
sp = {"buf": "", "tool": False}
|
||||
hold = len(BOX_START) - 1
|
||||
raw = []
|
||||
def emit_tools(chunk):
|
||||
raw.append(chunk)
|
||||
if dbg_echo:
|
||||
sys.stderr.write(chunk); sys.stderr.flush()
|
||||
if sp["tool"]:
|
||||
return
|
||||
sp["buf"] += chunk
|
||||
cut = sp["buf"].find(BOX_START)
|
||||
if cut >= 0:
|
||||
if cut:
|
||||
emit(sp["buf"][:cut])
|
||||
sp["buf"] = ""
|
||||
sp["tool"] = True
|
||||
return
|
||||
flush = max(0, len(sp["buf"]) - hold)
|
||||
if flush:
|
||||
emit(sp["buf"][:flush])
|
||||
sp["buf"] = sp["buf"][flush:]
|
||||
stats = self.server.engine.generate(
|
||||
prompt, maximum, temperature, top_p, emit, cache_slot)
|
||||
prompt, maximum, temperature, top_p, emit_tools, cache_slot)
|
||||
if not sp["tool"] and sp["buf"]:
|
||||
emit(sp["buf"]) # no tool call happened: flush held tail
|
||||
_content, calls = parse_tool_calls("".join(raw), tools)
|
||||
for i, tc in enumerate(calls):
|
||||
event([{"index": 0, "delta": {"tool_calls": [{"index": i, "id": tc["id"],
|
||||
"type": "function", "function": {"name": tc["function"]["name"],
|
||||
"arguments": tc["function"]["arguments"]}}]},
|
||||
"logprobs": None, "finish_reason": None}])
|
||||
finish = "tool_calls" if calls else ("length" if stats["length_limited"] else "stop")
|
||||
else:
|
||||
def emit_plain(chunk):
|
||||
if dbg_echo:
|
||||
sys.stderr.write(chunk); sys.stderr.flush()
|
||||
emit(chunk)
|
||||
stats = self.server.engine.generate(
|
||||
prompt, maximum, temperature, top_p, emit_plain, cache_slot)
|
||||
finish = "length" if stats["length_limited"] else "stop"
|
||||
ka_stop.set() # generation done: stop the keepalive pump
|
||||
ka_thread.join(timeout=2)
|
||||
final_choice = ({"index": 0, "delta": {}, "logprobs": None, "finish_reason": finish}
|
||||
if chat else {"index": 0, "text": "", "logprobs": None,
|
||||
"finish_reason": finish})
|
||||
@@ -535,10 +727,17 @@ class APIHandler(BaseHTTPRequestHandler):
|
||||
if reasoning_effort not in efforts:
|
||||
raise APIError(400, "`reasoning_effort` must be none, minimal, low, medium, high, or xhigh.",
|
||||
"reasoning_effort")
|
||||
# COLI_THINK=1 makes thinking the default when the client sends NEITHER reasoning_effort
|
||||
# nor enable_thinking (a global switch, like the old server's --think). An explicit
|
||||
# client value always wins. Default off => exact OpenAI-standard behavior.
|
||||
if (reasoning_effort is None and "enable_thinking" not in body
|
||||
and os.environ.get("COLI_THINK", "0") == "1"):
|
||||
reasoning_effort = "high"
|
||||
enable_thinking = body.get("enable_thinking", reasoning_effort not in (None, "none"))
|
||||
if not isinstance(enable_thinking, bool):
|
||||
raise APIError(400, "`enable_thinking` must be a boolean.", "enable_thinking")
|
||||
prompt = render_chat(body.get("messages"), enable_thinking, reasoning_effort)
|
||||
tools = body.get("tools") or body.get("functions") or None
|
||||
prompt = render_chat(body.get("messages"), enable_thinking, reasoning_effort, tools)
|
||||
self.generation(body, prompt, request_id, True)
|
||||
|
||||
def completion(self, body, request_id):
|
||||
|
||||
Reference in New Issue
Block a user