#6380 [Engine] Conversation cache: 0 reused on a single GPU since 0.2.11 (host layer); two-GPU layer split unaffected; engine version independent
open · @wl368 · 2 Kommentare · Auf GitHub
Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quants
Beschreibung
## Environment - Machine A (broken): 1x RTX 3090 24 GB, i9-13900, 64 GB RAM, Ubuntu 22.04.5, driver 580.173.02, CUDA 13 - Machine B (working): 2x RTX 4070 12 GB (layer split), i7-13700, Ubuntu 22.04, same driver - Model: agentionai/Qwen3.8-Flash-Next-Gyro-abliterated-Huihui-GGUF Gyro-M (abliterated file hardlinked to the official Gyro-M filename), context 262K, --kv int8, DFlash2, --vision cpu, --parallel 1 ## Symptom On the single-GPU machine, a second request with an identical 16K-token prefix is fully re-prefilled every time: ``` strata serve: prompt 16264 tokens = 0 reused + 16264 read in 6305 ms ... strata serve: conversation cache: parked 16343 tokens in 66.6 ms; parked=4 evictions=0 strata serve: prompt 16264 tokens = 0 reused + 16264 read in 6249 ms ... ``` Turn 2 with the same prefix takes full prefill time (~6.3 s) instead of sub-second. On 0.2.10 the same machine logged `156648 reused + 27 read` and answered follow-ups in 0.2-0.6 s. ## Bisect (machine A unless noted) | Terminal | Engine | Identical-prefix turn 2 | |---|---|---| | 0.2.10 | 0.1.56 | WORKS, 0.57 s | | 0.2.16 | 0.1.56 | broken, 8.9 s | | 0.2.16 | 0.1.57 / 0.1.58 / 0.1.59 | broken | | 0.2.16 | 0.1.60 / 0.1.62 | broken | | 0.2.17 | 0.1.63 | broken, 7.2 s | | 0.2.16 | 0.1.62 (machine B, 2 GPUs, layer split) | WORKS, 0.89 s | The regression tracks the Terminal/Control layer (introduced somewhere in 0.2.11+), not the engine version, and only appears on a single GPU. ## Ruled out - Fresh launch without the saved preset; without --kv-resident; STRATA_ROUTE_TAIL_SKIP=0; forcing mmap-experts; clean restart with parked=2-4 and evictions=0 (still 0 reused) - Engine args are byte-identical between the working 0.2.10 run and the broken 0.2.16 run on machine A, except expert-profile.bin now coming from components/control/0.1.62/data instead of terminal/0.1.56/data - Sometimes the parked conversation is spilled to the SSD tier; reuse still does not happen when it is parked in RAM only (evictions=0) - Cannot run the old Terminal on top of the new install: the CLI entry forwards to the newest installed Control, and the signed feed is monotonic ## Reproduce 1. Single NVIDIA GPU: `strata launch --family gyro --model Gyro-M --context 262k --parallel 1` 2. Send request A (16K-token prompt), then request B with the same prompt and a different final question 3. Engine log for B shows `0 reused`; wall time of B is close to A Happy to attach full engine logs and the launch config if useful. Thanks for the great project!
Mehr auf der Site
Links zu Install, Modellen, Releases.