Issues / #2090

#2090 [0.1.43] One GPU, default flags: "lending the prompt path its slots failed: ... do not fit" when a request's chunk is smaller than the auto chunk, engine exits (code 1)

open · @K8nsch · 2 comments · View on GitHub

BenchmarksSetup & installNVIDIA / CUDAModels & quantsWindowsLinux

Description

**Setup.** RTX 4070 Ti 12 GB (sm_89), driver 617.14, Windows 11 Pro 26200 (WDDM, the card also drives the
display), Ryzen 9 5900X, 64 GB DDR4. Release engine 0.1.43 (CUDA 13.0 zip, `BUILD.json` source release), server at
`7695859a`. Model Qwen3.8-Flash-Next IQ3_S native pack, setup's config plus `--calibrate`:

```
--pack iq3_s --native ...IQ3_S-00001-of-00002.gguf --ple-gguf ...IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin
--expert-cache auto --prefill auto --spec 4 --mtp rt --max-context 131072 --kv int8 --vision --vram-reserve-mib 700
--pcie-frac 0.35 --spec-min-p 0.70 --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97
```
(`"vision": {"gpu": false}`, one request at a time, no `--batch`.)

**What happens.** `tools/strata_bench.py --long` (the new BENCH.bat) failed in 2 of 9 runs. In both, a fresh
prompt whose chunk is *smaller than the auto chunk chosen at start* fails, the engine exits with code 1, and the
next request restarts it:

```
[strata] the engine reported an error: lending the prompt path its slots failed: prefill: device buffers for a chunk of 4096 tokens do not fit (34 of 12281 MiB free at the failure; 2589 MiB taken, the failing buffer wanted 40 MiB)
[strata] the engine stopped unexpectedly (exit code 1). ...
```

Free VRAM is not the limit: in the second case 436 MiB were free and the buffer wanted 30 MiB
(`... chunk of 3072 tokens do not fit (436 of 12281 MiB free at the failure; 2207 MiB taken, the failing buffer
wanted 30 MiB)`). Reading `Prefill::init` (`src/prefill/prefill.cpp`, the `carve(T, &o)` failure), the take that
fails is `used + bytes > cap` inside the borrowed region, i.e. the lent slots are smaller than the relayout for the
request's own chunk needs.

**The start-up sizing decides it** (every engine start of the day, from `strata-iq3_s.log`; the start-up free VRAM
was the same 3.82 GiB each time, the expert cache settles differently after "shrinking"):

| Start | Expert cache | Prompt chunk auto | Prompt path borrows | VRAM free, all loaded | Result |
|---|---|---|---|---|---|
| bench 1 | 1515 slots (no shrink line) | 5120, 199-slot ring | 1316 slots (2.53 GiB) | 180 MiB, "LOW" | 4K prompt (chunk 4096) fails, twice (after the restart the same sizing) |
| bench 1, 3rd start | 1418 slots | 4864 | 1266 slots (2.44 GiB) | 282 MiB | rest of the suite passes |
| `--vram-reserve-mib 1000` | 1292 slots | 4096 | 1117 slots (2.16 GiB) | 582 MiB | agent turn of 3,080 new tokens (chunk 3072) fails, twice |
| 7 other runs | 1414-1447 slots | 4864 | 1266 slots | 280-282 MiB | all pass, including chunks of 4096 and 3072 |
| `--expert-cache 1515` | 1628 slots | 5888 | 1463 slots (2.81 GiB) | 0 MiB, "LOW" | passes (prompts 1230 tok/s instead of 1830) |

So the failing pairs are auto 5120 -> request chunk 4096 and auto 4096 -> request chunk 3072, while auto 4864 ->
4096 / 3072 works. A larger `--vram-reserve-mib` does not help (it only moves the auto chunk). Disabling
`--adapt-*` did not change it either (0 failures in 2 runs without, 0 in 3 runs with).

**Expected.** A request chunk below the auto chunk fits in the region lent for the auto chunk (or the engine
falls back without exiting).

**Repro.** Not deterministic for me. Forcing the chunk does not trigger it, because an explicit chunk is checked at
start: with `--prefill 5120` the log says `prompt chunk 5120 -> 2560 tokens so its buffers fit in every expert
cache`, and fresh prompts of ~4,000 / 3,000 / 2,000 tokens (2 each) all pass. It only happened with
`--prefill auto`, and only when the auto chunk came out other than 4864 (5120 after a cache with no shrink step,
4096 with a 1000 MiB reserve). Maybe the same "fits in every expert cache" check is not applied to the auto chunk
against the request chunk's relayout?

Not run: other models, a second card, Linux.

Benchmark reports (scrubbed by the tool) can be attached on request.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.