#2650 Long-session decode drift on WSL2 + RTX 3090 24 GB at 262K: 59.3 → 80.0 tok/s in the last third; --kv-grow + reserve 1216 cuts the drift from −29% to −4.5%

Descrição

**Strata version, GPU, OS:** engine 0.1.44.1, source build (CUDA 13.4, `sm_86`) · RTX 3090 24 GB · **WSL2** Ubuntu 26.04 on Windows, host 128 GB RAM, `.wslconfig` `memory=96GB swap=0` · AMD Threadripper PRO 3975WX (AVX2, no AVX-512)

Model: Qwen3.8-Flash-Next GSQ-RCO **IQ3_S**, `--max-context 262144 --kv int8 --expert-profile --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp … --vision --vram-reserve-mib 700`, plus the experimental speed projection on (identical in both arms). One slot, no `--batch`.

This is the long-soak arm `docs/TEST_REQUESTS.md` asks for ("an hour or more of back-to-back requests … shows drift"), and it found drift.

## What happened

One engine start at 17:30, back-to-back agent traffic (multi-turn tool use, contexts 10K–80K), 124 requests with the engine's own per-request `decode_tok_s`, `hit_rate` and draft counts. Splitting that run into thirds by request order:

| | decode tok/s | expert hit | draft accept | mean context |
|---|---|---|---|---|
| first third | 84.0 | 83.4% | 78.9% | 47,461 |
| middle third | 77.2 | 84.1% | 79.3% | 69,512 |
| last third | **59.3** | 83.1% | 74.2% | **35,765** |

Whole run: mean 73.4, median 76.3, p10 55.5, min 40.5.

The control that matters: **hit rate stayed flat** (83.4% → 84.1% → 83.1%) while decode fell 29%, and the last third ran at a *shorter* mean context than the first. So this is not content, not cache warming, and not context length.

It is not clocks either: at load the card sat at 1890 MHz / 69–70 °C / 210–287 W against a 350 W limit, and `SW Thermal Slowdown`, `HW Thermal Slowdown`, `HW Power Brake` and `SW Power Cap` were all `Not Active` in `nvidia-smi -q -d PERFORMANCE`.

The tell is the engine's own line, which escalated across the first six requests:

```
strata serve: 274 MiB of VRAM free with everything loaded
strata serve: WARNING: only 274 MiB of VRAM free at the start of this request (below 512 MiB): … (#2065). Add --vram-reserve-mib 960 …
strata serve: WARNING: only 140 MiB of VRAM free … Add --vram-reserve-mib 1088 …
strata serve: WARNING: only   0 MiB of VRAM free … Add --vram-reserve-mib 1216 …
```

The sizer had priced the whole 262K K/V up front: `expert cache auto: 13.85 GiB free, 700 MiB reserved … -> 5224 slots`, then after shrinking `expert cache 6180 slots, 11.73 GiB of VRAM`, leaving 274 MiB on the card before any request ran.

## The change, and what it did

Two flags only, everything else byte-identical, second engine start at 21:39:

```
--kv int8 --kv-grow --vram-reserve-mib 1216      # was: --kv int8 --vram-reserve-mib 700
```

At start: `elastic K/V: 16384 of 262144 cells in VRAM (0.25 of 3.35 GiB); the expert cache (7411 slots) gives the K/V room below slot 5222 as the context grows`.

Both arms below are the **whole session** — all 124 requests before, all 54 after, no windowing:

| | before (n=124) | after (n=54) |
|---|---|---|
| expert cache | 6,180 slots / 12,012 MiB | **7,411 slots / 14,388 MiB** (+20%) |
| `vram_free_mib` at load | 274 | 803 |
| free during a request | 0–140 MiB | 587–871 MiB |
| decode mean / median | 73.4 / 76.3 | **82.8 / 81.4** |
| decode p10 / min | 55.5 / 40.5 | **73.4 / 66.4** |
| prefill aggregate (Σ read ÷ Σ prompt_ms) | 1,291 tok/s | **1,559 tok/s** |
| expert hit / draft accept | 83.6% / 77.4% | 82.6% / 73.3% |
| mean context | 50,791 | 73,102 |

Whole run, and the same split:

| | decode tok/s | expert hit | draft accept | mean context |
|---|---|---|---|---|
| first third | 83.8 | 80.6% | 75.5% | 44,520 |
| middle third | 84.5 | 82.9% | 76.4% | 62,617 |
| last third | 80.0 | 84.4% | 68.1% | **112,170** |

**The drift is much smaller, but it is not gone.** The before-run lost 29% from its first third to its last; the after-run loses 4.5%, and its last third runs at a **2.5× longer** mean context (112,170 vs 44,520) than its first — where the before-run's last third was at a *shorter* context. So the change bought a large level improvement and a much flatter shape, not a flat line.

Note the control now cuts the other way: hit rate and draft acceptance are **lower** after the change (83.6%→82.6%, 77.4%→73.3%) in exchange for +13% decode *and* +44% mean context. Fewer expert hits and fewer accepted drafts, still faster — so the gain is not an easier traffic mix.

## What `--kv-grow` costs, measured

During the steady climb the K/V takes ~52–57 slots per 8,192-cell step (`K/V grown to 81920 cells (1.07 GiB); the expert cache gave 56 slots for it … 6919 of 7411 slots hold experts`), and it gives them back when a session ends (`K/V grown to 16384 cells` again). It is not uniform, though: the *first* climb through a region is far more expensive than the second. Growing to 65,536 cells cost **326 slots** the first time and 56 the second, and 122,880 cells cost **218 slots** in a single move — so the cache re-learns which experts it can afford as the window moves. The cache also compacts as it goes (`7355 of 7411 slots hold experts` after a give-back). The cost is a slope, not a cliff:

| context | n | decode mean |
|---|---|---|
| 5–30K | 4 | 86.7 |
| 30–55K | 7 | 81.3 |
| 55–70K | 20 | 86.8 |
| 70–90K | 11 | 81.1 |
| 90K+ | 11 | 78.1 |

At 70–90K the `#2065` warning fires again (`only 414 MiB free`, `only 136 MiB free`) even with the 1216 MiB reserve, and decode settles at 78.1 above 90K — which is the same region the last third of the table above is measuring.

## Questions

1. **Should `--expert-cache auto` price the `--max-context` reservation differently on a card this full?** Setup wrote 262,144 with the whole-K/V default on a 24 GB card and left 274 MiB free at load. `--kv-grow`'s preconditions (one GPU, expert profile, no KV streaming, all experts in RAM) were all already true here — would it be worth defaulting it on, or having setup pick it, when they hold and the whole-K/V sizing lands under ~1 GiB free?
2. **The warning's suggested reserve climbs per request** (960 → 1088 → 1216). Could the start-up sizer begin at the value it would end up asking for, instead of discovering it over the first six requests?
3. **WSL2 has no fallback here.** `--kv-resident` is not usable (the WSL NVIDIA driver pins ~1 GB, as `docs/DETAILS.md` notes), so `--kv-grow` is the only path on WSL2 + 24 GB + a long context. Worth a line in the WSL notes, and maybe a setup warning for that combination?
4. **Is the 70–90K dip expected from ~55 slots per 8,192 cells**, or is there a cheaper place to take it from? The first-pass steps (326 slots at 65,536 cells, 218 at 122,880) look like a different regime from the steady ~55 — is re-learning the cache on the first climb through a region avoidable?

## Limits, stated plainly

- **The two arms are different traffic windows, not an interleaved A/B**, and the after-run is shorter (54 requests vs 124) while carrying a **longer** mean context (73,102 vs 50,791) — so the raw means are not a like-for-like speedup. The case for the change is the shape of the thirds plus the hit-rate/acceptance control above.
- **The client is the same engine** (this session serves the agent that ran the analysis), the same caveat as #1986.
- **Drift is reduced, not eliminated** — the after-run's last third is still 4.5% below its first, and decode keeps falling above 90K (78.1 mean). One engine session per arm; no multi-day number.
- Real agent traffic, not `BENCH.bat`: no fresh-prompt sweep, no needle recall, no quality check, no concurrency. Nothing about correctness is claimed.
- No cold-boot numbers; the OS page cache was warm in both arms. `--vision` was on in both — I did not test the vision-on-CPU change from #2406, which looks relevant on a card this full.

## Reproducing

Numbers come from the engine's own lines (`prompt … generated in … tok/s, drafts accepted …`, `hit rate: …`, `K/V grown to …`) and `GET /metrics`, never from dividing tokens by wall time. The parsing script is two regexes over `strata-<model>.log`, split at `engine started`; happy to add it as a `tools/` script or a `bench/results/` folder if that is the right shape.