#5943 [Engine] --prefill auto keeps a 5250-slot lend region on a fully-resident 3x 32 GB split (0.1.58 kept 0): decode -26%, prefill -46%; STRATA_PREFILL_LEND_PCT=0 restores the full tier

open · @noahark · 4 コメント · GitHub で見る

BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows

本文

## Area

[Engine] — speed, the server

## Desktop app, Terminal or source?

Source: a sm_70 build of v0.2.13.1-160 (engine 0.1.59.1, CUDA 12.4) on Windows, started as `serve/server.py --engine strata --config <run config>.json` (the launcher wraps it). The comparison engine is a sm_70 build of v0.2.12-4 (engine 0.1.58) run the same way, same run config.

## What happened

On a 3x Tesla V100-PCIE-32GB layer split (16/16/16, Qwen3.8-Flash-Next 125B UD-IQ4_XS requant, `--kv int8 --kv-resident 32768 --expert-cache auto --prefill auto --prefill-pipe 1536`, side-loaded Q8_0 engram via `--ple-gguf`), 0.1.58 plans **no loan** and fills the whole expert set into the GPU cache:

```
strata serve:   CUDA0 keeps its own prompt buffers (4.17 GiB of 10.56 GiB free): no loan
strata generate: expert cache auto: 28.14 GiB free, 2000 MiB reserved (...) -> 8192 slots
strata generate: layer split, CUDA1: 29.43 GiB free of 31.86, room for experts 26.98 GiB
/metrics: expert_slots 24576, expert_cache_mib 57587
```

0.1.59.1, same config, same model, same pack: the prefill plan keeps **5250 lend slots**, the GPU cache plans 40.8 GiB and later re-balances to 26.1 GiB with a 39.7 GiB RAM copy, and the tier the engine settles on is **18166 of 24576 slots (74%)**:

```
strata: prefill plan inputs: 56.2 GiB experts, 40.8 GiB in the GPU caches, 27.4 GiB RAM copy (5250 lend slots kept), ...
strata generate: layer split, CUDA1: 6.73 GiB free of 31.86, room for experts 4.28 GiB
/metrics: expert_slots 18166, expert_cache_mib 41767
strata serve: resident RAM: 27.41 GiB of experts in RAM, 4542 exchanged with the VRAM tier, 0 blob reads from the file   <- count keeps climbing
strata serve: decode expert cache hit rate: 85.0-94.3% per request (was: every request fully in VRAM)
```

Measured cost (same machine, same run config, same bench, 3 rounds per band, server-side per-request timings; each arm a fresh load):

| arm | engine | expert slots | short decode | 12K prefill | long decode |
|---|---|---|---|---|---|
| 1 | 0.1.58 | **24576** | 113.5 tok/s | 2100 tok/s | 103.1 tok/s |
| 2 | 0.1.59.1 | **18166** | 84.4 tok/s | 1124 tok/s | 88.6 tok/s |
| 3 | 0.1.59.1 + `STRATA_PREFILL_LEND_PCT=0` | **24576** | 117.4 tok/s | 1857 tok/s | 104.6 tok/s |

Arm 3 restores the 0.1.58 plan shape exactly:

```
strata generate: expert cache auto: 27.88 GiB free, 2000 MiB reserved (...) -> 8192 slots
strata generate: layer split, CUDA1: 29.17 GiB free of 31.86, room for experts 26.72 GiB
```

Draft acceptance rates return to the arm-1 band too (78-83% vs the 85-90% the thrashing arm-2 showed).

So the lend region is the mechanism: it is sized for cards too small to hold their working set (the 12 GB card measurement in the code comment), but on a shape whose whole expert set fits in VRAM - and where 0.1.58's planner already said "no loan" - the reservation only trades guaranteed VRAM hits for prompt-buffer headroom the shape does not use, and every miss then streams over PCIe. This looks like the same lever as #5199 (16 GB card, default chunk lending 3.5 GB out of the slots) and #5462 (per-card cache falls as cards are added), applied through the `--prefill auto` plan rather than the chunk default.

## Smallest steps or command that reproduce it

Three arms on one machine, same model, same pack, same run config; only the engine binary (and `STRATA_PREFILL_LEND_PCT` in arm 3) changes. Each arm: fresh load, then 3x short prompt (~86 tokens, 320 max tokens) and 3x 11.8K-token prompt (200 max tokens), interleaved rounds. Compare `/metrics` `expert_slots` and the bands above. The plan lines quoted above are the short version: arm 2 says "room for experts 4.28 GiB" where arms 1 and 3 say ~26 GiB.

## Reproduction status

Reproduced locally (all three arms above, this rig).

## Strata version

Strata v0.2.13.1-160 · engine 0.1.59.1 (sm_70 source build) — comparison arm: Strata v0.2.12-4 · engine 0.1.58 (sm_70 source build).

## Graphics card, memory and system

3x Tesla V100-PCIE-32GB (32 GB each, TCC) · 128 GB RAM (DDR3L-1600 ECC REG) · Windows 10 Enterprise LTSC 2021 (19044), driver 581.80

## Model, size and settings

Qwen3.8-Flash-Next 125B, Unsloth UD-IQ4_XS requant (4 shards) + side-loaded Q8_0 engram via `--ple-gguf` · 262144 context, `--kv int8 --kv-resident 32768` · pictures on (`--vision`)

関連リンク

インストール・モデル・リリースへのサイト内リンク。