coli plan: read-only Disk/RAM/VRAM placement planner (mirrors engine budget math, versioned --json, --auto-tier) (#27)
* Add tiered resource planner * Apply resource plans to runner configuration --------- Co-authored-by: JustVugg <JustVugg@users.noreply.github.com>
This commit is contained in:
@@ -91,6 +91,25 @@ cd c
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli chat
|
||||
```
|
||||
|
||||
Inspect the planned storage hierarchy before loading the model:
|
||||
|
||||
```bash
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli plan
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48 --json
|
||||
|
||||
# apply the bounded plan to the normal runner
|
||||
COLI_MODEL=/nvme/glm52_i4 ./coli chat --auto-tier
|
||||
```
|
||||
|
||||
`coli plan` reads only safetensors headers and reports the model's exact dense/expert
|
||||
footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Its
|
||||
versioned JSON output is intended to be shared by the CLI, API server, Web UI, and
|
||||
desktop shell; it does not allocate model tensors or start inference.
|
||||
`--auto-tier` applies the same plan to `chat`, `run`, `serve`, and benchmarks. It
|
||||
sets the RAM budget and context immediately; the VRAM tier is enabled only when
|
||||
the current `glm` binary is linked with CUDA. Explicit flags and environment
|
||||
variables keep precedence over automatic values.
|
||||
|
||||
The engine at runtime is pure C — python is only used by the one-time converter.
|
||||
|
||||
### OpenAI-compatible API
|
||||
|
||||
Reference in New Issue
Block a user