Issues / #1951

#1951 serve: keep-alive is an SSE comment, so clients that drop comments still hit their stream idle timeout and retry

open · @rebuilt · 1 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPModels & quants

Descripción

## Summary

The server's streaming keep-alive (the prefill/quiet heartbeat) is written as an **SSE comment**
(`: ping`). At least one real client — OpenCode — applies its stream idle timeout to the *parsed SSE
frame* stream and **drops comments before that accounting**, so a long prefill with a heartbeat
flowing still trips the client's 300 s `chunkTimeout`. The client aborts a healthy request and retries,
and each retry starts a new prompt read: the classic retry/re-prefill loop.

Sending a **real SSE frame** instead of a comment fixes it at the source, for every client.

This is a correction/extension of #1619 and #1666: both suggest a `: keepalive` / `ping` comment. That
keeps the socket warm and notices a dead client, but it does **not** reset the idle timeout of a client
that drops comments before its timeout accounting.

## Environment

- Strata `serve/server.py` (server) + engine **0.1.39**, backend `hip`, AMD Radeon AI PRO R9700 (gfx1201)
- Model Qwen3.8-Flash-Next IQ3_S, `--max-context 131072`, `--kv int8 --kv-resident 32768`, `--prefill 4096 --spec 4`, `--mmap-experts`
- Client: OpenCode (`opencode serve --service`), OpenAI-compatible provider over SSE at `http://127.0.0.1:12434/v1`, default `chunkTimeout` 300 s
- Trigger: long-context prompts (72k–100k tokens) whose prefill takes 1–4 minutes

## What we see

The client records mid-stream transport errors with HTTP 200 (so the response had started):

```
type: provider.transport   status: 200
  "Timed out waiting for response data"          # the stream idle/chunk timeout
  "Connection lost while reading the response"   # the same abort, generic path
```

and retries, each retry re-reading the prompt. The engine log shows the abandoned turn:

```
strata serve: prompt 72822 tokens = 0 reused + 4096 of 72822 read in 17866 ms (229.3 tok/s),
  0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 0 checkpoints (cancelled)
```

## Root cause

`serve/server.py` writes the heartbeat as an SSE comment:

```
: keepalive
```

The client's idle timeout is applied to the parsed SSE **frame** stream, and its parser drops comment
lines before that accounting. A steady trickle of comments therefore never resets the timeout, and a
prefill longer than the client's timeout is aborted no matter how healthy the stream is. (Verified
against OpenCode: its `chunkTimeout` is applied to parsed frames.)

## Fix

Emit a real frame at the HTTP writer level (so it covers both chunk generators uniformly):

- **OpenAI** (`/v1/chat/completions`): an empty `chat.completion.chunk`, reusing the last real chunk's
  `id` / `object` / `created`/`model`:

  ```
  data: {"id": "...","object":"chat.completion.chunk","created":...,"model":"...","choices":[{"index":0,"delta":{},"finish_reason":null}]}
  ```

- **Anthropic** (`/v1/messages`):

  ```
  event: ping
  data: {"type": "ping"}
  ```

- `/v1/responses` already sends real `response.*` frames; unchanged.

Keeping the last real chunk's `id`/`model` avoids re-triggering parsers that key on a stable chunk id.

## Tests

New `KeepAliveFrames` in `serve/test_server.py`:

- `test_chat_keepalive_is_a_frame`
- `test_anthropic_keepalive_is_a_ping_event`

## Verification

After the change, live for ~5.5 h of long-context traffic on the same rig:

- **0** client retries / `"Timed out waiting for response data"` / `"Connection lost while reading the response"`
- 301 requests served, 0 stalls, 0 engine faults, 0 server errors
- chat stream carries real `"delta": {}` frames, **0 SSE comments**, ends with `data: [DONE]`; Anthropic emits `event: ping`
- long contexts complete with a warm cache, e.g. `prompt 96632 tokens = 95673 reused + 959 read`

## Related

- #1619 (queue-wait sends no bytes) — same class of bug; **please use real frames, not `: keepalive` comments**, or clients that drop comments will still time out.
- #1666 (keep-alive during tool-call argument generation) — same note.

En el sitio

Enlaces a install, modelos, releases.