问题 / #431
#431 Cancelling a streaming request during a long prompt read only takes effect after the full prompt is read
closed · @jkuepker · 1 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIAMD / HIPModels & quantsLinux
说明
With `"stream": true`, a disconnect is detected (`cancel=True`, 0 tokens generated), but only after the whole prompt has been read. On a 57K-token prompt, the next request waited about 17 s. **Repro:** 1. Stream a 57,396-token request and disconnect after 6 s: ```bash curl -s -N -m 6 127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @long-57k-stream.json ``` 2. Immediately send "Say OK." (max_tokens 30). **Observed:** ``` strata serve: prompt 57396 tokens = 0 reused + 57396 read in 22151 ms (2591.2 tok/s), 0 generated ..., 2 checkpoints (cancelled) [strata] done: 0 tokens in 22 s (0.0 tok/s) (disconnect, cancel=True) ``` The follow-up request took **17.2 s**, even though it only needed about 0.9 s of engine time. `serve/server.py` already checks `cancel` on every `PP` progress line. So either the engine doesn't act on STOP between prefill chunks, or `--prefill auto` makes the chunks so large that the first check comes late. I haven't checked which. **Expected:** a cancel during the prompt read stops it at the next chunk boundary. The release notes for 0.1.32 list "a cancelled long prompt followed by a new request" as checked. That claim holds: the new request is answered correctly with no engine restart. This issue is only about how long the new request waits. **Environment** ``` Strata v0.1.32 (tag c499bd1), Linux, Ubuntu 24.04.4, kernel 7.0.0-34 GPU: Radeon AI PRO R9700 32 GB (gfx1201), single card CPU: i5-12600K (AVX2, no AVX-512), 62 GB DDR5 Setup: ./setup.sh --backend hip --family coder --model IQ1_M (no system ROCm) ROCm: TheRock 7.10.0a20251120 wheels (gfx120X-all) installed by setup into .venv; hipBLASLt 1.2.0 Server: serve/server.py with setup's generated config (max-context 131072, kv int8, spec 4 + MTP) ```
本站相关内容
相关页面的快捷入口。