#6171 [Engine] 0.2.15 window warm-up with STRATA_ROUTE_PRIOR: the first request after a start often ends in the middle of a number

closed · @baptisterajaut · 1 comentários · No GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Descrição

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

Since 0.2.15, the first request after an engine start ends with `<|im_end|>` in the middle of a long number far more often than later requests. With `STRATA_WINDOW_WARM=0`, or without `STRATA_ROUTE_PRIOR`, it goes back to the 0.2.13.1 level. Measured with a 60K-token passkey prompt (fresh nonce, no prefix-cache hit), 8 engine starts per arm, logprobs on: the probability of `<|im_end|>` summed over the 8 answer digits has a median of 1.0 % on 0.2.13.1, 4.8 to 7.4 % on 0.2.15 (two sessions), 1.7 % on 0.2.15 with `STRATA_WINDOW_WARM=0`, and 1.4 % on 0.2.15 with the prior off (1.5 % with both off). Across every 0.2.15 arm that kept both the warm-up and the prior, 10 of 137 answers came out short (`'6797748'`, `'34425'`, `'2193'`…), against 0 of 48 in the arms without one of them and 0 of 16 on 0.2.13.1.

The warm-up comment (`generate.cpp`, "window warm-up (lane DRAFT-COSTS)") says it changes nothing a request can see except that the expert cache may page in what the dummy token routes to, "speed only". With the route prior that part is no longer speed only, since the prior moves routing towards what is resident. My reading, not verified in the code path: the dummy windows fill the cache with the experts of token id 1, the prior pulls the first request towards them, and the adaptive tier cleans it up over the next requests, which would explain why an established engine is fine.

Ruled out on the same box and day: building with `-DSTRATA_MMQ=ON` (6.7 %), `STRATA_PREFILL_CPU_SHARE=0` (6.6 %), `STRATA_ADAPT_NOWAIT=1` and `--adapt-async 0` (6.1 and 6.5 %), and #5655 cherry-picked on 0.2.15 (6.9 %), so it is not the sentinel write that #5655 fixes.

Possible directions (yours to judge): skip the warm-up, or its cache page-in, when the route prior is on, or put the cache back in the state it had before the dummy windows. We run with `STRATA_WINDOW_WARM=0` for now.

### Smallest steps or command that reproduce it

Start the engine with `STRATA_ROUTE_PRIOR=1.5` and DFlash2, send one 60K-token cold prompt that asks for an 8-digit number hidden in it, with `logprobs: true, top_logprobs: 5`, read P(`<|im_end|>`) on each digit, stop the engine, repeat. Eight starts are enough to separate the arms; a single request is not, the per-answer values have a long tail (1 to 50 %) in both versions.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.15 · engine 0.1.61, plus two local patches (a `--ple-gguf` loading fix and a prefill CPU-share token cap, `STRATA_PREFILL_CPU_TOKENS`; removing the second does not change the result, 6.6 %). Not run on plain v0.2.15.

### Graphics card, memory and system

RTX 3090 24 GB on PCIe 3.0 x16 · 2x Xeon Gold 6252, 187 GB RAM · Debian 12, kernel 6.1, driver 620.70.06, CUDA 13.2, sm_86 build

### Model, size and settings

Qwen3.8-Flash-Next IQ4_XS, every expert in the arena (no `--resident-budget-gib`), `--draft dflash2 --spec 6 --draft-conf 0.5`, `--kv int8`, 262K context, `--adapt-async 1`, `STRATA_ROUTE_PRIOR=1.5`, server under `numactl --interleave=all`. Pictures on.

### Logs

The warm-up line at start: `strata serve: window warm-up: ms per width 1:18 2:20 3:21 4:23 5:28 6:32 7:36 8:39, total 371 ms`. No error or non-finite line in any of the runs.

Measured and prepared with Claude Code on my box; the numbers are from its logs.

No site

Links install, modelos, releases.