#6403 [Engine] 12 GB of VRAM on an RTX 5080: --prefill auto picks a 7,936-token chunk where 8,192 fits; 16K prompts read in three pieces, 16-30 % slower
open · @Retarel · 0 Kommentare · Auf GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Beschreibung
### Area
Engine
### Desktop app, Terminal or source?
Terminal (`strata serve --profile <copy of the qwen-128k launch config>`), release packages.
### What happened
With about 12 GB of VRAM left to the engine (the rest kept free with `--vram-reserve-mib`, as when another program shares the card), `--prefill auto` settles on a **7,936-token** chunk. An explicit `--prefill 16384` on the same config is cut back to **8,192** ("prompt chunk 16384 -> 8192 tokens so its buffers fit in every expert cache"), and that start also ends up with more expert-cache slots (2,581 against 2,432). So 8,192 fits, but the automatic pick stays 256 tokens below it, and a 16K prompt then needs three pieces instead of two.
The difference is largest right there, and it shows at every length up to 64K:
Fresh (uncacheable) prompts, prompt read in tok/s, engine 0.1.63; 16 GB and 12 GB auto are the mean of 2 engine starts, 12 GB with `--prefill 16384` is 1 start:
| prompt | 16 GB, auto (17,664) | 12 GB, auto (7,936) | 12 GB, `--prefill 16384` (-> 8,192) |
|---|---|---|---|
| 8K | 3,211 | 3,051 | 3,115 |
| 16K | 4,460 | 2,991 | **3,474 (+16 % vs auto)** |
| 32K | 4,673 | 3,224 | 3,563 (+11 %) |
| 64K | 4,769 | 3,388 | 3,602 (+6 %) |
| 100K | 4,574 | 3,421 | 3,430 (0 %) |
standard-v1 on the same starts:
| | 16 GB, auto | 12 GB, auto | 12 GB, `--prefill 16384` |
|---|---|---|---|
| sanity | 12/12 | 12/12 | 12/12 |
| prompt-4k read / decode | 2,364 / 89.1 | 2,358 / 72.9 | 2,285 / 86.0 |
| prompt-16k read / decode | 4,560 / 83.0 | **2,709** / 83.0 | **3,517** / 79.0 |
| short decode | 96.6 | 85.4 | 83.0 |
Russian decode (5 prompts) is the same in the two 12 GB arms (74-127 tok/s against 76-127). Engine lines, 12 GB:
```text
auto: strata generate: expert cache 2432 slots, 5.47 GiB of VRAM
strata serve: prompt chunk auto: 7936 tokens, a 384-slot ring
strata serve: the prompt path borrows 2170 CUDA0 cache slots (4.88 GiB)
--prefill 16384: strata generate: expert cache 2581 slots, 5.81 GiB of VRAM
strata serve: prompt chunk 16384 -> 8192 tokens so its buffers fit in every expert cache
strata serve: the prompt path borrows 2227 CUDA0 cache slots (5.02 GiB)
```
`--prefill auto:16384` (a ceiling) changes nothing at 12 GB: the same 7,936 and the same 2,432 slots.
Suggestion: when the fit lands just under a multiple of 8,192 (7,936 here), take the 8,192 that the explicit path shows to fit, so a 16,384-token prompt stays two pieces.
**Side notes from the same session, for the 0.2.17 notes and #2065:**
- 0.2.17 against 0.2.14.1 on this card at 16 GB (qwen-128k, DFlash2, 2 starts per version): standard-v1 prompt-4k 1,859 (0.2.14.1, a run on this PC on 11-16) -> 2,364 tok/s (+27 %; the CPU prompt share following the link: `PCIe share: ... the 28.9 GB/s link ... pcie_frac 0.25`), 16K-100K +4-6 %, decode level, 12/12.
- The new `STRATA_ADAPT_FETCH=2` default on a 16 GB sm_120 card, OrcaRouter Qwen3.8-Flash-Next-Uncensored IQ3_XXS (third-party pack), Russian decode, one start each: 0.2.17 default 104 / 124 / 131 / 144 / 168 tok/s (short / prose / code / JSON / edit) against `STRATA_ADAPT_FETCH=0` on 0.2.17 82 / 118 / 118 / 121 / 125, and 0.2.14.1 75 / 115 / 109 / 124 / 119.
- One start with a web browser holding VRAM (`only 238 MiB of VRAM free at the start of this request`) read prompts 15-25 % slower than the next start of the same config with 491 MiB free (8K 2,790 against 3,212, 16K 3,337 against 4,322, 32K 3,465 against 4,460); with the browser closed (646 MiB free) the numbers matched the second start.
### Smallest steps or command that reproduce it
On a 16 GB card, start a Flash-Next UD-IQ4_XS run config with `--vram-reserve-mib` set so the engine takes ~12.3 GB (2,944 here), once with `--prefill auto` and once with `--prefill 16384`, and read fresh 16K-64K prompts.
### Reproduction status
Reproduced: 2 starts with `--prefill auto`, 1 with `--prefill 16384`, 1 with `--prefill auto:16384`.
### Strata version
Terminal 0.2.17, Control and engine 0.1.63 (windows-x86_64-cuda-cuda13).
### Graphics card, memory and system
RTX 5080 16 GB (sm_120, PCIe 5.0 x8), Ryzen 9 9950X3D, 96 GB DDR5, Windows 11 Pro, page file 64 GB fixed.
### Model, size and settings
Unsloth UD-IQ4_XS (the 09-30 files: gate/up IQ3_S, down IQ4_NL / Q8_0), `--max-context 131072 --kv int8 --kv-resident 32768 --expert-cache auto --spec 4 --spec-min-p 0.5`, DFlash2, `--mtp-draft-vocab` cyrillic, `STRATA_ADAPT_FETCH=2`, `--vram-reserve-mib 2944` (engine 12,264-12,328 MiB by nvidia-smi against the desktop's ~950 MiB).
Mehr auf der Site
Links zu Install, Modellen, Releases.