Pull requests / #1953
#1953 serve: the keep-alive while a prompt is read is a real SSE frame, not a comment
open · @rebuilt · 0 commentaires · Sur GitHub
BenchmarksServer & APIModels & quants
Description
## What
The streaming keep-alive (the heartbeat while a long prompt is read, or while a tool argument is held back) is written as an **SSE comment** (`: keep-alive`). A client that resets its stream idle timeout on parsed SSE **frames** drops comments before that accounting, so a slow prefill times out and the client retries from scratch every time.
This changes the two writers whose clients time out this way:
- **OpenAI** `/v1/chat/completions`: an empty `chat.completion.chunk`, reusing the last real chunk's `id`/`created`/`model` (`choices: [{index: 0, delta: {}, finish_reason: null}]`).
- **Anthropic** `/v1/messages`: a real `event: ping` / `data: {"type":"ping"}`.
`/v1/responses` already sends a real `response.in_progress` event every 15 s, so it is unchanged.
## Why not the comment
Reported from a live OpenCode session against a local Strata (gfx1201, IQ3_S 128k): 72k–100k-token prompts, 1–4 minute prefills. OpenCode applies its `chunkTimeout` (default 300 s) to the parsed frame stream and ignores comments, so it aborted mid-stream with `"Timed out waiting for response data"` / `"Connection lost while reading the response"` and retried, each retry re-reading the prompt. The engine log showed the abandoned turn:
```
strata serve: prompt 72822 tokens = 0 reused + 4096 of 72822 read in 17866 ms (229.3 tok/s), 0 generated in 0 ms ... (cancelled)
```
After the change, ~5.5 h of the same traffic: **0** client timeouts/retries, 301 requests served, 0 stalls, 0 engine faults.
## Related
- #1619 (a queued request sends no bytes) and #1666 (a held-back tool argument) suggest the same comment. The queue wait and the held-back argument still send real frames now; please use frames, not comments, in those paths too — a comment will not reset a comment-dropping client.
- Fixes the client-side half of #1951.
## Tests
- New `serve/test_server.py::KeepAliveFrames` — `test_chat_keepalive_is_a_frame`, `test_anthropic_keepalive_is_a_ping_event`.
- `PromptProgress.test_a_client_that_does_not_ask_gets_a_keep_alive_frame` now expects the empty chunk instead of the comment.
Sur le site
Liens install, modèles, releases.