ZacharyZcR cbd599024e Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68) ...

* Fuse CUDA expert MLP execution

* Group CUDA expert transfers by device

* Instrument grouped CUDA expert execution

* Bound grouped CUDA decode scratch

* Execute expert groups across GPUs in parallel

* Release host backing for multi-GPU experts

* Define quality-preserving memory policies

* Overlap cold expert loading with resident compute

* Adapt expert placement with session LFRU

* Fuse q4 expert gate and up dispatch

* Plan CPU work on physical cores

* Batch grouped expert CUDA kernels

* Separate VRAM and RAM expert placement

* Add ragged multi-sequence decode forward

* feat(runtime): add continuous decode scheduler

* Route concurrent API requests through batch scheduler

* Harden multiplex request lifecycle and framing

* Cancel disconnected multiplex requests

* Bind API port before starting the engine

* fix automatic KV slot allocation

* add native int4 Tensor Core grouped GEMM

* add Tensor Core throughput benchmark

* optimize packed int4 low-row kernels

* add asynchronous CUDA staging streams

* document validated six-GPU dense acceleration

* tune six-GPU expert hot set

* raise validated expert hot-set target

* add CUDA MLA absorption core

* fuse grouped expert gate and up projections

* Warn for explicit lossy routing flags

2026-07-13 14:30:36 +02:00

..

scripts

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

tests

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

tools

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

.gitignore

Windows 11 native port, phase 1: MinGW-w64 static build, compat shims, setup + docs (#40)

2026-07-11 12:59:49 +02:00

backend_cuda.cu

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

backend_cuda.h

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

backend_metal.h

Metal backend (Apple Silicon): batched experts + fused attention on GPU, unified-memory zero-copy, gated behind COLI_METAL — 2.06 tok/s M5 Max (#72, #87, #103)

2026-07-13 08:57:10 +02:00

backend_metal.mm

Metal backend (Apple Silicon): batched experts + fused attention on GPU, unified-memory zero-copy, gated behind COLI_METAL — 2.06 tok/s M5 Max (#72, #87, #103)

2026-07-13 08:57:10 +02:00

coli

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

compat.h

Windows 11 native port, phase 1: MinGW-w64 static build, compat shims, setup + docs (#40)

2026-07-11 12:59:49 +02:00

decode_batch.h

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

doctor.py

coli doctor: read-only setup/health diagnostics (path, config, shards, disk, RAM budget, placement) (#33)

2026-07-11 19:39:13 +02:00

glm.c

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

grammar.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

iobench.c

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

json.h

colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts

2026-07-05 20:57:25 +02:00

Makefile

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

olmoe.c

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

openai_server.py

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

ref_glm.json

colibrì: pure-C GLM-5.2 (744B MoE) engine with disk-streamed experts

2026-07-05 20:57:25 +02:00

resource_plan.py

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

setup.sh

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

st.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

tier.h

Unify continuous batching + heterogeneous runtime: decode batching, physical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (#68)

2026-07-13 14:30:36 +02:00

tok_unicode.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00

tok.h

Translate user-facing runtime output to English, machine prefixes preserved, + CLI output test (#67, #85)

2026-07-12 13:38:40 +02:00