Isolated sequence KV contexts: KVState extraction, KV_SLOTS serve contexts with per-slot persistence, budget-aware pool accounting (#29)

* Add isolated sequence KV contexts

* Log projected multi-slot KV footprint
This commit is contained in:
ZacharyZcR
2026-07-10 22:19:43 +08:00
committed by GitHub
parent 76e858a80f
commit 9e6e1ac327
5 changed files with 192 additions and 67 deletions
+21
View File
@@ -159,6 +159,27 @@ requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent
`GET /health` exposes active/queued/completed/rejected counters, and successful
generation responses include `x-colibri-queue-wait-ms`.
### Isolated KV contexts
`coli serve --kv-slots N` allocates up to 16 independent sequence contexts. Requests
select one with the optional integer `cache_slot` field; ordinary OpenAI clients omit
it and keep the original slot 0 behavior.
```json
{
"model": "glm-5.2-colibri",
"messages": [{"role": "user", "content": "Continue this conversation"}],
"cache_slot": 1
}
```
Each slot owns its token history, compressed MLA/DSA KV memory, MTP window, and
crash-safe persistence file (`.coli_kv`, `.coli_kv.1`, ...). The engine still executes
one sequence at a time; this establishes explicit KV ownership without pretending that
threaded HTTP is continuous batching. RAM admission accounts for every configured slot.
Use `COLI_KV_SLOTS=N` as the environment equivalent. Start with a small value: at the
default 4096-token context, every slot costs hundreds of MB.
### Experimental resident CUDA backend
colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming