danielkrause/colibri-strix
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bounded FIFO inference scheduling: admission queue, OpenAI-shaped 429/503, queued-disconnect detection (#28)

Browse Source
This commit is contained in:
ZacharyZcR
2026-07-10 18:52:07 +08:00
committed by GitHub
parent 3cbc52ab00
commit 76e858a80f
4 changed files with 345 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8
View File
@@ -151,6 +151,14 @@ Browser access from the Vite development server and Tauri local origins is enabl
by default. Repeat `--cors-origin https://your-ui.example` to allow another exact
origin, or use `--cors-origin '*'` only on a trusted local network.
The engine owns one mutable KV context, so HTTP generation uses a bounded FIFO
admission queue instead of pretending to run unsafe parallel sequences. Configure it
with `--max-queue N` (default 8) and `--queue-timeout SECONDS` (default 300), or the
`COLI_MAX_QUEUE` / `COLI_QUEUE_TIMEOUT` environment variables. Saturated and timed-out
requests receive OpenAI-shaped HTTP 429 errors before streaming headers are sent.
`GET /health` exposes active/queued/completed/rejected counters, and successful
generation responses include `x-colibri-queue-wait-ms`.
### Experimental resident CUDA backend
colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming