Windows native port: serve-mode pipe fix + RAM detection + POSIX guards, AVX-VNNI kernel, gated CUDA DLL (#131, fixes #123)
Rebased onto current dev, split into 3 logical parts (all validated): 1. CPU portability (serve-mode _O_BINARY pipe fix — stock main hangs on MinGW without it; RAM detection cap 0->9/layer; POSIX guards for select/mmap/madvise; warmup script). 2. AVX-VNNI 128-bit int8/int4 dot kernel (Alder Lake+/Meteor Lake+), bit-identical to AVX2 (author-verified on Meteor Lake; compiles out to AVX2 elsewhere) + _mm256_extracti128_si256 typo fix that blocked -march=native. 3. CUDA DLL via LoadLibrary, gated behind CUDA_DLL=1 (host never links cudart; silent CPU fallback if absent; author-verified on RTX 5070 Ti). Validated here: make check 59/59, oracle 32/32 TF, Windows cross-compile clean + glm.exe loads+runs via WSL interop. Fixes the #123 Windows build failure.
This commit is contained in:
@@ -160,6 +160,12 @@ make iobench.exe # disk I/O benchmark
|
||||
make test-c # run C tests
|
||||
make test-python # run Python tests (requires python)
|
||||
|
||||
# AVX-VNNI: Intel Alder Lake+ (and Meteor Lake+) CPUs have a 128-bit int8
|
||||
# dot-product instruction (VPDPBUSD) the engine can use for ~1.3x faster
|
||||
# quantized matmul. The x86-64-v3 default (portable AVX2) compiles it out;
|
||||
# build for THIS machine to enable it:
|
||||
make glm.exe ARCH=native # banner prints "idot: avx-vnni"
|
||||
|
||||
# Verify (tiny model, 2.4 MB):
|
||||
pip install torch transformers safetensors huggingface_hub
|
||||
python tools/make_glm_oracle.py # generate tiny oracle
|
||||
@@ -171,9 +177,49 @@ python coli chat --model D:\glm52_i4 # interactive chat
|
||||
python coli serve --model D:\glm52_i4 # OpenAI-compatible API
|
||||
```
|
||||
|
||||
**Status:** Phase 1 complete (compiles, correct, static-linked). O_DIRECT (Phase 2),
|
||||
GPU via `LoadLibrary` on `coli_cuda.dll` (Phases G0–G2), and full-model validation
|
||||
are separate workstreams. See `PORT_WINDOWS_PLAN.md` for the full plan.
|
||||
**Warmup (overnight cache priming):** the engine's expert cache learns from
|
||||
your workload. The included `warmup.ps1` script runs `coli run` in a loop with
|
||||
diverse prompts to build the `.coli_usage` histogram unattended, so the next
|
||||
real session starts with a large, accurate hot-expert pin. Each run saves usage
|
||||
atomically on clean completion.
|
||||
|
||||
```powershell
|
||||
.\warmup.ps1 -Rounds 1 -Ngen 32 # ~60-90 min, durable progress
|
||||
```
|
||||
|
||||
**NVIDIA GPU (optional, via runtime DLL):** on Windows the engine is built with
|
||||
MinGW gcc but CUDA kernels require MSVC + nvcc. The split is clean: build the
|
||||
CUDA backend into a standalone `coli_cuda.dll` (nvcc + MSVC), then the host
|
||||
`glm.exe` loads it at runtime via `LoadLibrary` (`c/backend_loader.c`). The host
|
||||
never links cudart directly; if the DLL is absent the engine falls back to CPU
|
||||
without error.
|
||||
|
||||
```powershell
|
||||
# Prerequisites: CUDA Toolkit + MSVC Build Tools (cl.exe) + nvcc on PATH.
|
||||
# Build the DLL from a shell with the MSVC environment set (vcvars64.bat or
|
||||
# "x64 Native Tools Command Prompt for VS"):
|
||||
make cuda-dll CUDA_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8" CUDA_ARCH=sm_120
|
||||
|
||||
# Build the host with the runtime loader (CUDA_DLL=1 adds -DCOLI_CUDA and
|
||||
# links backend_loader.o instead of cudart):
|
||||
make glm.exe CUDA_DLL=1 ARCH=native
|
||||
|
||||
# Run with the GPU expert tier (8 GB VRAM budget here; scale to your free VRAM):
|
||||
$env:COLI_CUDA="1"; $env:COLI_GPU="0"; $env:CUDA_EXPERT_GB="8"
|
||||
python coli chat --model D:\glm52_i4 --topp 0.7
|
||||
```
|
||||
|
||||
The DLL exports 11 `extern "C"` symbols (`coli_cuda_init`, `coli_cuda_matmul`,
|
||||
etc.); `backend_loader.c` resolves them via `GetProcAddress` on first use.
|
||||
`ColiCudaTensor*` is opaque to the host (stored, never dereferenced), so the
|
||||
MSVC-allocated struct is safe across the ABI boundary. `CUDA_ARCH` must match
|
||||
your GPU's compute capability (e.g. `sm_120` for Blackwell / RTX 50-series,
|
||||
`sm_89` for Ada / RTX 40-series).
|
||||
|
||||
**Status:** Phase 1 complete (compiles, correct, static-linked). The Windows
|
||||
GPU tier (runtime `coli_cuda.dll` via `LoadLibrary`) is implemented and
|
||||
verified on RTX 50-series (sm_120). O_DIRECT (Phase 2) and full-model
|
||||
validation against the transformers oracle remain separate workstreams.
|
||||
|
||||
### OpenAI-compatible API
|
||||
|
||||
|
||||
Reference in New Issue
Block a user