#6082 [Engine] PLE residency policy locks the n-gram table on a 96 GB Windows PC, then the #2090 guard cuts the prompt chunk 21,248 -> 8,192: prompts 9-14% slower, conversation cache off
open · @Retarel · 1 コメント · GitHub で見る
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
本文
### Area
Engine
### Desktop app, Terminal or source?
Terminal (`strata serve --profile <run config>`), release packages.
### What happened
On a 96 GB Windows PC the PLE residency policy locks the 26.8 GiB n-gram table in RAM because table + expert arena + headroom fit (`ple_resident_admit`), but what that leaves is then too little for two other automatic rules: the #2090 RAM guard steps the prompt chunk down from 21,248 to 8,192, and the automatic conversation cache gets 2.8 GiB or turns itself off. On this machine the result is slower than the direct reads the policy replaces: prompts 9-14% slower, start 22 s longer, no parked conversations for an agent. Decode is the same either way.
The policy's choice also sits on the edge here: 84.9 GiB available (the engine's figure) admitted the table, about 84.0 GiB free before a start (a browser open in another program) kept the direct reads, so the same run config starts in either mode depending on what else is open.
A/B on one engine (0.1.60, Control 0.1.60.1), same run config, only the table mode differs, a fresh engine start per row, nothing else running except where noted, 15 s pause between starts:
| run | table | start, s | RAM free after load | prompt chunk | conversation cache | prompt 16K / 32K / 64K, tok/s | decode, median of 5 Russian prompts |
|---|---|---|---|---|---|---|---|
| 1 | direct (the policy's own choice; 84.0 GiB free before the start, a browser open elsewhere) | 26 | 25.9 GiB | 21,248 | 8,192 MiB | 4,066 / 4,265 / 4,569 | 115.7 |
| 2 | direct (`STRATA_PLE_AUTO_RAM=0`) | 24 | 28.6 GiB | 21,248 | 8,192 MiB | 4,200 / 4,561 / 4,606 | 118.1 |
| 3 | `--ple-io ram` | 47 | 2.8 GiB | 8,192 | off ("not enough free RAM to park conversations") | 3,680 / 4,114 / 4,161 | 117.5 |
| 4 | direct (`STRATA_PLE_AUTO_RAM=0`) | 26 | 29.0 GiB | 21,248 | 8,192 MiB | 4,216 / 4,540 / 4,568 | 119.1 |
| 5 | `--ple-io ram`, 85.9 GiB free before the start | 49 | 3.0 GiB | 8,192 | off | 3,614 / 4,031 / 4,136 | 121.3 |
Earlier the same day, the policy's own choice (no `--ple-io` given, 84.9 GiB available) on 0.1.59 and on 0.1.60 gave the RAM-mode row: table locked in 22.4 / 22.5 s, chunk 8,192, and after "PLE table lock given back" 2.8 GiB for the conversation cache.
The table on direct reads keeps up on this SSD: the start-up probe says 118,000-146,000 rows/s ("fine").
The code comment in `ple_residency.hpp` measures the lock at 2-3% of prompt speed, and #4518 already gives the lock back when that makes the conversation cache bigger. Here the loss comes from the prompt chunk: the #2090 guard (`nv_chunk.hpp`) sees 4,241-4,527 MiB available after the table and the arena and keeps 8,192, while the same PC on direct reads gets 21,248 (its lines: 16,384 would hold 2,305 MiB more in RAM during a prompt, 32,768 another 4,600 MiB). On a card that lends cache slots to a 21K chunk, that costs more than the lock saves.
Suggestion: count the prompt-chunk loan the #2090 guard would allow, and the automatic conversation cache's minimum, in the headroom `ple_resident_admit` checks, so the table is locked only when it does not push either of them down. Failing that, a log line when the lock costs the chunk step would make it visible.
### Smallest steps or command that reproduce it
Start a model whose experts are about 50 GiB on a 96 GB PC with ~85 GiB available, once with `--ple-io ram` (or the policy's own choice) and once with `STRATA_PLE_AUTO_RAM=0`, then read fresh 16K-64K prompts.
### Reproduction status
Reproduced: 2 starts per mode on the same day, plus the two earlier policy-chosen starts.
### Strata version
Terminal 0.2.14.1, Control 0.1.60.1, engine 0.1.60 (windows-x86_64-cuda-cuda13). The policy-chosen RAM mode was also seen on engine 0.1.59.
### Graphics card, memory and system
RTX 5080 16 GB (sm_120, PCIe 5.0 x8), Ryzen 9 9950X3D, 96 GB DDR5 (95.6 GB visible), Windows 11 Pro, page file 64 GB fixed, NVMe SSD.
### Model, size and settings
OrcaRouter Qwen3.8-Flash-Next-Uncensored IQ3_XXS (third-party GGUF, packed with `iq_pack.py --compat-bf16`; expert arena 49.8 GiB, n-gram table 26.8 GiB), hand-written run config: `--max-context 262144 --kv int8 --kv-resident 32768 --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt> --mtp-draft-vocab cyrillic --vram-reserve-mib 1472 --vision`, `vram_elastic`, env `STRATA_HC_Q8=1 STRATA_ROUTE_TAIL_SKIP=0`. The Unsloth UD-IQ4_XS run config on the same PC keeps the direct reads (its experts are larger, so the table never fits).
### Logs
RAM mode chosen by the policy:
```text
strata generate: --ple-io ram chosen by the residency policy (the table is 26.8 GiB, the expert arena still to fill 49.8 GiB and 84.9 GiB of RAM is available; STRATA_PLE_AUTO_RAM=0 keeps the direct reads, --ple-io direct forces them)
strata generate: PLE table locked in RAM (--ple-io ram chosen by the residency policy) in 22.5 s
strata generate: NVIDIA: prompt chunks stay at 8192: 16384 would lend 1111 more cache slots (2305 MiB more held in RAM during a prompt), and 4316 MiB of RAM is available for 0 MiB of resident experts plus that (#2090; STRATA_NV_CHUNK=1 takes the default ceiling anyway)
strata serve: prompt chunk auto: 8192 tokens, a 512-slot ring
strata serve: PLE table lock given back: the lock left 2.8 GiB of RAM for the conversation cache, 2.8 GiB without it (the table stays cached; --conversation-cache-mib N and --ple-io ram are kept as asked)
```
Direct reads:
```text
strata generate: PLE direct read probe: 121118 rows/s (fine)
strata serve: prompt chunk auto: 21248 tokens, a 512-slot ring
strata serve: conversation cache sized from the free RAM: 8192 MiB for up to 4 parked conversations, so an agent's next turn reuses its prompt (--conversation-cache-mib N sets it, 0 turns it off)
```
関連リンク
インストール・モデル・リリースへのサイト内リンク。