#6417 [Engine] 0.1.63: after one picture through the elastic image encoder, the expert cache can grow back a segment short and --prefill auto:32768 sticks at 16384 (60K cold prompt -37 %)

open · @baptisterajaut · 0 comments · View on GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

After one picture through the elastic GPU image encoder, the expert cache can grow back one segment short, and `--prefill auto:32768` then stays at 16384 until the next start. A 60K-token cold prompt reads at 2,764 tok/s right after the start and at 1,730 tok/s after a single picture, same engine session, passkey answered both times (-37 %).

At start the 32768 chunk borrows 3,093 of the 3,449 cache slots (`the prompt path borrows 3093 CUDA0 cache slots`), 89.7 % against the 90 % `kAutoLendPct` cap, so it holds only while the cache keeps 3,437 slots. The picture shrinks the cache (`VRAM 2048`), the encoder's idle stop grows it back (`VRAM 0`), and `ExpertCache::grow` maps whole segments only: `VRAM 0 MiB kept free: the expert cache 3030 -> 3394 of 3449 slots (14.00 of 14.23 GiB), 221 MiB free now, prompt chunk 16384`. One segment is 55 slots (~232 MiB) here, and `relend()` judges the cap on the 3,394 live slots, so 32768 is refused and 16384 sticks. It does not happen on every cycle: on another start the same picture brought the cache back to 3,449 slots with 3 MiB free. The free VRAM at the shrink went 1,883 -> 1,771 -> 1,713 MiB over a session, so something seems to keep a little VRAM each cycle; my guess would be the verify window graphs captured while the cache is shrunk (`capturing the 5-token window (388 MiB of VRAM free)`), not traced.

With `STRATA_PREFILL_LEND_PCT=92` and one segment taken off on purpose (`POST /v1/vram {"reserve_mib": 400}`, 3,394 slots), the chunk stays at 32768 and the same prompt reads at 2,930 tok/s; on a fresh start it reads at 2,717, so the setting changes nothing until the cache is short. We run with it now.

Possible directions (yours to judge): `relend()` could keep the start's chunk when the loan still leaves `g_lend_keep` slots and the shortfall against the cap is under one segment, or the cap could be judged against `full_slots()` rather than the live count; or `grow` could map a partial last segment when the free VRAM falls a few MiB short of a whole one.

### Smallest steps or command that reproduce it

Start with `--vision` (GPU encoder, elastic), `--prefill auto:32768`, `--expert-cache auto`, on a card where the start's loan sits close to the 90 % cap (the `borrows N ... cache slots` line against the cache size). Send one picture, wait `elastic_idle_s` (120 s) for the encoder to stop, and read the last `VRAM 0 MiB kept free` line: when it shows fewer slots than the full count, the chunk shows 16384 and a long cold prompt reads in twice as many chunks.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.17 · engine 0.1.63, plus #6214's auto-mode pin and our opt-in `STRATA_ARENA_NUMA` patches (neither touches the cache or the prompt loan). Not run on plain v0.2.17.

### Graphics card, memory and system

RTX 3090 24 GB on PCIe 3.0 x16 · 2x Xeon Gold 6252, 187 GB RAM · Debian 12, kernel 6.1, driver 620.70.06, CUDA 13.2, sm_86 build

### Model, size and settings

Qwen3.8-Flash-Next Q6_K, every expert in the arena, `--prefill auto:32768`, `--expert-cache auto`, `--vram-reserve-mib 125`, `--draft dflash2 --spec 6 --draft-conf 0.5`, `--kv int8`, 262K context, `--adapt-async 1`, `--vision` with the GPU encoder, `STRATA_PREFILL_CPU_SHARE_NT=64`, `STRATA_PREFILL_CPU_SHARE_MAX=12288`.

### Logs

```
strata serve: prompt chunk auto: 32768 tokens, a 512-slot ring
strata serve: the prompt path borrows 3093 CUDA0 cache slots (12.76 GiB)
strata serve: prompt 60083 tokens = 0 reused + 60083 read in 21735 ms (2764.4 tok/s), …
strata serve: VRAM 2048 MiB kept free: the expert cache 3449 -> 3030 of 3449 slots (12.50 of 14.23 GiB), 1771 MiB free now, prompt chunk 16384, in 8 ms
strata verify: capturing the 5-token window (388 MiB of VRAM free)
strata serve: VRAM 0 MiB kept free: the expert cache 3030 -> 3394 of 3449 slots (14.00 of 14.23 GiB), 221 MiB free now, prompt chunk 16384, in 135 ms
strata serve: prompt 60084 tokens = 0 reused + 60084 read in 34727 ms (1730.2 tok/s), …
```

Related on this site

Quick links to related pages.