#6380 [Engine] Conversation cache: 0 reused on a single GPU since 0.2.11 (host layer); two-GPU layer split unaffected; engine version independent

open · @wl368 · 2 comentários · No GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quants

Descrição

## Environment
- Machine A (broken): 1x RTX 3090 24 GB, i9-13900, 64 GB RAM, Ubuntu 22.04.5, driver 580.173.02, CUDA 13
- Machine B (working): 2x RTX 4070 12 GB (layer split), i7-13700, Ubuntu 22.04, same driver
- Model: agentionai/Qwen3.8-Flash-Next-Gyro-abliterated-Huihui-GGUF Gyro-M (abliterated file hardlinked to the official Gyro-M filename), context 262K, --kv int8, DFlash2, --vision cpu, --parallel 1

## Symptom
On the single-GPU machine, a second request with an identical 16K-token prefix is fully re-prefilled every time:

```
strata serve: prompt 16264 tokens = 0 reused + 16264 read in 6305 ms ...
strata serve: conversation cache: parked 16343 tokens in 66.6 ms; parked=4 evictions=0
strata serve: prompt 16264 tokens = 0 reused + 16264 read in 6249 ms ...
```

Turn 2 with the same prefix takes full prefill time (~6.3 s) instead of sub-second. On 0.2.10 the same machine logged `156648 reused + 27 read` and answered follow-ups in 0.2-0.6 s.

## Bisect (machine A unless noted)
| Terminal | Engine | Identical-prefix turn 2 |
|---|---|---|
| 0.2.10 | 0.1.56 | WORKS, 0.57 s |
| 0.2.16 | 0.1.56 | broken, 8.9 s |
| 0.2.16 | 0.1.57 / 0.1.58 / 0.1.59 | broken |
| 0.2.16 | 0.1.60 / 0.1.62 | broken |
| 0.2.17 | 0.1.63 | broken, 7.2 s |
| 0.2.16 | 0.1.62 (machine B, 2 GPUs, layer split) | WORKS, 0.89 s |

The regression tracks the Terminal/Control layer (introduced somewhere in 0.2.11+), not the engine version, and only appears on a single GPU.

## Ruled out
- Fresh launch without the saved preset; without --kv-resident; STRATA_ROUTE_TAIL_SKIP=0; forcing mmap-experts; clean restart with parked=2-4 and evictions=0 (still 0 reused)
- Engine args are byte-identical between the working 0.2.10 run and the broken 0.2.16 run on machine A, except expert-profile.bin now coming from components/control/0.1.62/data instead of terminal/0.1.56/data
- Sometimes the parked conversation is spilled to the SSD tier; reuse still does not happen when it is parked in RAM only (evictions=0)
- Cannot run the old Terminal on top of the new install: the CLI entry forwards to the newest installed Control, and the signed feed is monotonic

## Reproduce
1. Single NVIDIA GPU: `strata launch --family gyro --model Gyro-M --context 262k --parallel 1`
2. Send request A (16K-token prompt), then request B with the same prompt and a different final question
3. Engine log for B shows `0 reused`; wall time of B is close to A

Happy to attach full engine logs and the launch config if useful. Thanks for the great project!

No site

Links install, modelos, releases.