Issues / #2316

#2316 [Bug] Repeated requests sharing a large prefix collapse the checkpoint replay path to 7-21 tok/s (cold prefill: 2100+); --prompt-cache 0 avoids it

open · @Jerry-Lee661 · 0 comments · View on GitHub

BenchmarksServer & APIAMD / HIPModels & quantsLinux

Description

## Environment
- Strata v0.1.43.1, Linux (Ubuntu 26.04.1), self-built for HIP
- 2× AMD Radeon RX 7900 XTX (gfx1100), layer split `auto`, dual-socket-free E5-2673 v3, 128 GB DDR3
- Model: Qwen3.8-Flash-Next (Swift-1.5 GSQ-RCO abliterated), native IQ3_S pack + experts.bin (48.4 GiB)
- Relevant flags: `--serve --ple-io ram --kv k8v4 --max-context 524288 --rope-scaling yarn --spec 3 --prefill auto --expert-profile <shipped> --expert-cache 4096`; prompt-cache left at defaults (`6`, every `16384`)

## Symptom
On the same running instance, re-submitting a request that shares a large (~48K tok) prefix with an earlier request collapses the effective prompt throughput to **7-21 tok/s** — vs **2050-2170 tok/s** for a cold prompt of identical length. Decode is unaffected. The first large prefill after a restart is always full speed; repeats of the same request collapse.

## Isolation (three experiments, same session)
1. 48K prompt, first (cold) = **2059 tok/s**. Same prompt again (cache hit) = **17.4 → 7.0 tok/s** on subsequent repeats.
2. A *different* 48K prompt with no shared prefix = **2155 tok/s** (full speed) → rules out a generic "second big prefill" state problem.
3. `--prompt-cache 0` + restart → three consecutive same-prefix 48K requests = **2111 / 2164 / 2172 tok/s**. Symptom gone.

This points at the conversation-checkpoint replay path: on a cache hit, the restore/replay of the cached prefix runs ~100-300× slower than a cold prefill of the same content.

## Notes
- The restore seems to be the slow part: `prompt_per_second` is computed over the non-cached suffix, and the wall time is dominated by the replay of the cached prefix.
- Possibly related to #1703 (long prompts get re-read from the beginning), but there the re-read runs at normal speed (~1684 tok/s in that report); here the cache-hit path itself is 100-300× slower than cold.
- Reproduces on default cache flags (`--prompt-cache 6`, `--prompt-cache-every 16384`).

## Workaround
`--prompt-cache 0` — fully removes the collapse. In-session incremental prefill still works; only idle-session checkpoint parking is lost.

## Platform numbers (same build, for context)
- 48K prefill: 2050-2172 tok/s; ~342K prefill: 1947 tok/s (cold, no shared prefix)
- Short-prompt decode: 43-47 tok/s; 300-token prose: 33-47 tok/s (45-47 with `STRATA_SPLIT_OVERLAP=1` from self-built #1120)
- NIAH-style retrieval correct at 325K / 417K / 502K input tokens

Happy to provide full command lines, logs, or run any probe on this machine.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.