Issues / #431

#431 Cancelling a streaming request during a long prompt read only takes effect after the full prompt is read

closed · @jkuepker · 1 comentarios · En GitHub

BenchmarksSetup & installServer & APIAMD / HIPModels & quantsLinux

Descripción

With `"stream": true`, a disconnect is detected (`cancel=True`, 0 tokens generated), but only after the whole prompt has been read. On a 57K-token prompt, the next request waited about 17 s.

**Repro:**
1. Stream a 57,396-token request and disconnect after 6 s:
   ```bash
   curl -s -N -m 6 127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @long-57k-stream.json
   ```
2. Immediately send "Say OK." (max_tokens 30).

**Observed:**
```
strata serve: prompt 57396 tokens = 0 reused + 57396 read in 22151 ms (2591.2 tok/s), 0 generated ..., 2 checkpoints (cancelled)
[strata] done: 0 tokens in 22 s (0.0 tok/s) (disconnect, cancel=True)
```
The follow-up request took **17.2 s**, even though it only needed about 0.9 s of engine time.

`serve/server.py` already checks `cancel` on every `PP` progress line. So either the engine doesn't act on STOP between prefill chunks, or `--prefill auto` makes the chunks so large that the first check comes late. I haven't checked which.

**Expected:** a cancel during the prompt read stops it at the next chunk boundary.

The release notes for 0.1.32 list "a cancelled long prompt followed by a new request" as checked. That claim holds: the new request is answered correctly with no engine restart. This issue is only about how long the new request waits.

**Environment**

```
Strata v0.1.32 (tag c499bd1), Linux, Ubuntu 24.04.4, kernel 7.0.0-34
GPU: Radeon AI PRO R9700 32 GB (gfx1201), single card
CPU: i5-12600K (AVX2, no AVX-512), 62 GB DDR5
Setup: ./setup.sh --backend hip --family coder --model IQ1_M (no system ROCm)
ROCm: TheRock 7.10.0a20251120 wheels (gfx120X-all) installed by setup into .venv; hipBLASLt 1.2.0
Server: serve/server.py with setup's generated config (max-context 131072, kv int8, spec 4 + MTP)
```

En el sitio

Enlaces a install, modelos, releases.