coli plan: read-only Disk/RAM/VRAM placement planner (mirrors engine budget math, versioned --json, --auto-tier) (#27)

* Add tiered resource planner

* Apply resource plans to runner configuration

---------

Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
This commit is contained in:
ZacharyZcR
2026-07-10 18:51:23 +08:00
committed by GitHub
parent 1453dab7ae
commit 3cbc52ab00
4 changed files with 412 additions and 4 deletions
+19
View File
@@ -91,6 +91,25 @@ cd c
COLI_MODEL=/nvme/glm52_i4 ./coli chat
```
Inspect the planned storage hierarchy before loading the model:
```bash
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json
# apply the bounded plan to the normal runner
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier
```
`coli plan` reads only safetensors headers and reports the model's exact dense/expert
footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Its
versioned JSON output is intended to be shared by the CLI, API server, Web UI, and
desktop shell; it does not allocate model tensors or start inference.
`--auto-tier` applies the same plan to `chat`, `run`, `serve`, and benchmarks. It
sets the RAM budget and context immediately; the VRAM tier is enabled only when
the current `glm` binary is linked with CUDA. Explicit flags and environment
variables keep precedence over automatic values.
The engine at runtime is pure C — python is only used by the one-time converter.
### OpenAI-compatible API