#5943 [Engine] --prefill auto keeps a 5250-slot lend region on a fully-resident 3x 32 GB split (0.1.58 kept 0): decode -26%, prefill -46%; STRATA_PREFILL_LEND_PCT=0 restores the full tier
open · @noahark · 4 commentaires · Sur GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Description
## Area [Engine] — speed, the server ## Desktop app, Terminal or source? Source: a sm_70 build of v0.2.13.1-160 (engine 0.1.59.1, CUDA 12.4) on Windows, started as `serve/server.py --engine strata --config <run config>.json` (the launcher wraps it). The comparison engine is a sm_70 build of v0.2.12-4 (engine 0.1.58) run the same way, same run config. ## What happened On a 3x Tesla V100-PCIE-32GB layer split (16/16/16, Qwen3.8-Flash-Next 125B UD-IQ4_XS requant, `--kv int8 --kv-resident 32768 --expert-cache auto --prefill auto --prefill-pipe 1536`, side-loaded Q8_0 engram via `--ple-gguf`), 0.1.58 plans **no loan** and fills the whole expert set into the GPU cache: ``` strata serve: CUDA0 keeps its own prompt buffers (4.17 GiB of 10.56 GiB free): no loan strata generate: expert cache auto: 28.14 GiB free, 2000 MiB reserved (...) -> 8192 slots strata generate: layer split, CUDA1: 29.43 GiB free of 31.86, room for experts 26.98 GiB /metrics: expert_slots 24576, expert_cache_mib 57587 ``` 0.1.59.1, same config, same model, same pack: the prefill plan keeps **5250 lend slots**, the GPU cache plans 40.8 GiB and later re-balances to 26.1 GiB with a 39.7 GiB RAM copy, and the tier the engine settles on is **18166 of 24576 slots (74%)**: ``` strata: prefill plan inputs: 56.2 GiB experts, 40.8 GiB in the GPU caches, 27.4 GiB RAM copy (5250 lend slots kept), ... strata generate: layer split, CUDA1: 6.73 GiB free of 31.86, room for experts 4.28 GiB /metrics: expert_slots 18166, expert_cache_mib 41767 strata serve: resident RAM: 27.41 GiB of experts in RAM, 4542 exchanged with the VRAM tier, 0 blob reads from the file <- count keeps climbing strata serve: decode expert cache hit rate: 85.0-94.3% per request (was: every request fully in VRAM) ``` Measured cost (same machine, same run config, same bench, 3 rounds per band, server-side per-request timings; each arm a fresh load): | arm | engine | expert slots | short decode | 12K prefill | long decode | |---|---|---|---|---|---| | 1 | 0.1.58 | **24576** | 113.5 tok/s | 2100 tok/s | 103.1 tok/s | | 2 | 0.1.59.1 | **18166** | 84.4 tok/s | 1124 tok/s | 88.6 tok/s | | 3 | 0.1.59.1 + `STRATA_PREFILL_LEND_PCT=0` | **24576** | 117.4 tok/s | 1857 tok/s | 104.6 tok/s | Arm 3 restores the 0.1.58 plan shape exactly: ``` strata generate: expert cache auto: 27.88 GiB free, 2000 MiB reserved (...) -> 8192 slots strata generate: layer split, CUDA1: 29.17 GiB free of 31.86, room for experts 26.72 GiB ``` Draft acceptance rates return to the arm-1 band too (78-83% vs the 85-90% the thrashing arm-2 showed). So the lend region is the mechanism: it is sized for cards too small to hold their working set (the 12 GB card measurement in the code comment), but on a shape whose whole expert set fits in VRAM - and where 0.1.58's planner already said "no loan" - the reservation only trades guaranteed VRAM hits for prompt-buffer headroom the shape does not use, and every miss then streams over PCIe. This looks like the same lever as #5199 (16 GB card, default chunk lending 3.5 GB out of the slots) and #5462 (per-card cache falls as cards are added), applied through the `--prefill auto` plan rather than the chunk default. ## Smallest steps or command that reproduce it Three arms on one machine, same model, same pack, same run config; only the engine binary (and `STRATA_PREFILL_LEND_PCT` in arm 3) changes. Each arm: fresh load, then 3x short prompt (~86 tokens, 320 max tokens) and 3x 11.8K-token prompt (200 max tokens), interleaved rounds. Compare `/metrics` `expert_slots` and the bands above. The plan lines quoted above are the short version: arm 2 says "room for experts 4.28 GiB" where arms 1 and 3 say ~26 GiB. ## Reproduction status Reproduced locally (all three arms above, this rig). ## Strata version Strata v0.2.13.1-160 · engine 0.1.59.1 (sm_70 source build) — comparison arm: Strata v0.2.12-4 · engine 0.1.58 (sm_70 source build). ## Graphics card, memory and system 3x Tesla V100-PCIE-32GB (32 GB each, TCC) · 128 GB RAM (DDR3L-1600 ECC REG) · Windows 10 Enterprise LTSC 2021 (19044), driver 581.80 ## Model, size and settings Qwen3.8-Flash-Next 125B, Unsloth UD-IQ4_XS requant (4 shards) + side-loaded Q8_0 engram via `--ple-gguf` · 262144 context, `--kv int8 --kv-resident 32768` · pictures on (`--vision`)
Sur le site
Liens install, modèles, releases.