问题 / #430

#430 serve: non-streaming request is not cancelled when the client disconnects; the engine runs to max_tokens and blocks the next request

closed · @jkuepker · 1 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIAMD / HIPModels & quantsLinux

说明

When a client drops a non-streaming `/v1/chat/completions` request (for example, a timeout), the engine keeps going. It reads the whole prompt and generates up to `max_tokens`. The next request waits for all of that to finish.

**Repro:**
1. Send a long non-streaming request and drop it after 4 s:
   ```bash
   curl -s -m 4 127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @long-57k.json   # max_tokens 300
   ```
2. Immediately send a small request ("Say OK.", max_tokens 30).

**Observed:** the engine log shows the dropped request ran to the end (41,012 tokens read in 30.6 s, then all 300 tokens generated). The small request took **33 s**.
```
strata serve: prompt 57396 tokens = 16384 reused + 41012 read in 30626 ms (1339.1 tok/s), 300 generated in 4673 ms (64.2 tok/s) ...
```

**Expected:** the request is cancelled soon after the disconnect, as streaming requests already are (`finish = "disconnect"`). One possible approach: while a non-streaming request runs, poll the client socket for EOF, for example with `select` and `MSG_PEEK` on each heartbeat or `PP` line, and set `cancel` when it closes.

This matters for clients with request timeouts, such as agents and proxies. A timed-out request keeps occupying the single engine.

**Environment**

```
Strata v0.1.32 (tag c499bd1), Linux, Ubuntu 24.04.4, kernel 7.0.0-34
GPU: Radeon AI PRO R9700 32 GB (gfx1201), single card
CPU: i5-12600K (AVX2, no AVX-512), 62 GB DDR5
Setup: ./setup.sh --backend hip --family coder --model IQ1_M (no system ROCm)
ROCm: TheRock 7.10.0a20251120 wheels (gfx120X-all) installed by setup into .venv; hipBLASLt 1.2.0
Server: serve/server.py with setup's generated config (max-context 131072, kv int8, spec 4 + MTP)
```

本站相关内容

相关页面的快捷入口。