15bf8cac6b
openai server: type tool arguments from schema + implement tool_choice, 9 new tool-calling tests ( #118 )
...
Two OpenAI-compat tool-calling bugs found against the real GLM-5.2 (dnnspaul): (1) string-typed args coerced to numbers — declared schema type now decides, string kept verbatim, bool rejected as number, schema-less params keep permissive decoding; (2) tool_choice was ignored — none/auto/required/{function} now honored, invalid returns 400. Python-only (openai_server.py + tests), engine untouched. 36/36 tests pass (verified independently in a clean worktree).
2026-07-13 16:11:31 +02:00
cbd599024e
Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) ( #68 )
...
* Fuse CUDA expert MLP execution
* Group CUDA expert transfers by device
* Instrument grouped CUDA expert execution
* Bound grouped CUDA decode scratch
* Execute expert groups across GPUs in parallel
* Release host backing for multi-GPU experts
* Define quality-preserving memory policies
* Overlap cold expert loading with resident compute
* Adapt expert placement with session LFRU
* Fuse q4 expert gate and up dispatch
* Plan CPU work on physical cores
* Batch grouped expert CUDA kernels
* Separate VRAM and RAM expert placement
* Add ragged multi-sequence decode forward
* feat(runtime): add continuous decode scheduler
* Route concurrent API requests through batch scheduler
* Harden multiplex request lifecycle and framing
* Cancel disconnected multiplex requests
* Bind API port before starting the engine
* fix automatic KV slot allocation
* add native int4 Tensor Core grouped GEMM
* add Tensor Core throughput benchmark
* optimize packed int4 low-row kernels
* add asynchronous CUDA staging streams
* document validated six-GPU dense acceleration
* tune six-GPU expert hot set
* raise validated expert hot-set target
* add CUDA MLA absorption core
* fuse grouped expert gate and up projections
* Warn for explicit lossy routing flags
2026-07-13 14:30:36 +02:00
9e6e1ac327
Isolated sequence KV contexts: KVState extraction, KV_SLOTS serve contexts with per-slot persistence, budget-aware pool accounting ( #29 )
...
* Add isolated sequence KV contexts
* Log projected multi-slot KV footprint
2026-07-10 16:19:43 +02:00
76e858a80f
Bounded FIFO inference scheduling: admission queue, OpenAI-shaped 429/503, queued-disconnect detection ( #28 )
2026-07-10 12:52:07 +02:00
3e4d08b6bf
OpenAI-compatible HTTP API: stdlib-only gateway over SERVE with KV prefix reuse across stateless requests ( #21 )
...
* Add OpenAI-compatible HTTP API
* Support browser API clients
* Handle missing KV cache during rewind
2026-07-10 11:04:56 +02:00
Dennis Paul