Issues / #2316
#2316 [Bug] Repeated requests sharing a large prefix collapse the checkpoint replay path to 7-21 tok/s (cold prefill: 2100+); --prompt-cache 0 avoids it
open · @Jerry-Lee661 · 0 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPModels & quantsLinux
Beschreibung
## Environment - Strata v0.1.43.1, Linux (Ubuntu 26.04.1), self-built for HIP - 2× AMD Radeon RX 7900 XTX (gfx1100), layer split `auto`, dual-socket-free E5-2673 v3, 128 GB DDR3 - Model: Qwen3.8-Flash-Next (Swift-1.5 GSQ-RCO abliterated), native IQ3_S pack + experts.bin (48.4 GiB) - Relevant flags: `--serve --ple-io ram --kv k8v4 --max-context 524288 --rope-scaling yarn --spec 3 --prefill auto --expert-profile <shipped> --expert-cache 4096`; prompt-cache left at defaults (`6`, every `16384`) ## Symptom On the same running instance, re-submitting a request that shares a large (~48K tok) prefix with an earlier request collapses the effective prompt throughput to **7-21 tok/s** — vs **2050-2170 tok/s** for a cold prompt of identical length. Decode is unaffected. The first large prefill after a restart is always full speed; repeats of the same request collapse. ## Isolation (three experiments, same session) 1. 48K prompt, first (cold) = **2059 tok/s**. Same prompt again (cache hit) = **17.4 → 7.0 tok/s** on subsequent repeats. 2. A *different* 48K prompt with no shared prefix = **2155 tok/s** (full speed) → rules out a generic "second big prefill" state problem. 3. `--prompt-cache 0` + restart → three consecutive same-prefix 48K requests = **2111 / 2164 / 2172 tok/s**. Symptom gone. This points at the conversation-checkpoint replay path: on a cache hit, the restore/replay of the cached prefix runs ~100-300× slower than a cold prefill of the same content. ## Notes - The restore seems to be the slow part: `prompt_per_second` is computed over the non-cached suffix, and the wall time is dominated by the replay of the cached prefix. - Possibly related to #1703 (long prompts get re-read from the beginning), but there the re-read runs at normal speed (~1684 tok/s in that report); here the cache-hit path itself is 100-300× slower than cold. - Reproduces on default cache flags (`--prompt-cache 6`, `--prompt-cache-every 16384`). ## Workaround `--prompt-cache 0` — fully removes the collapse. In-session incremental prefill still works; only idle-session checkpoint parking is lost. ## Platform numbers (same build, for context) - 48K prefill: 2050-2172 tok/s; ~342K prefill: 1947 tok/s (cold, no shared prefix) - Short-prompt decode: 43-47 tok/s; 300-token prose: 33-47 tok/s (45-47 with `STRATA_SPLIT_OVERLAP=1` from self-built #1120) - NIAH-style retrieval correct at 325K / 417K / 502K input tokens Happy to provide full command lines, logs, or run any probe on this machine.
Mehr auf der Site
Links zu Install, Modellen, Releases.