Pull requests / #1673

#1673 serve: heartbeat a request that waits for the turn (#1619)

open · @nekomario28 · 0 comentarios · En GitHub

Server & API

Descripción

Fixes #1619.

With the default single request turn a queued request received **no bytes at all**: the fifo was taken with a blocking acquire before the response's first yield. A client with a stream idle timeout aborts a healthy queued request, each retry starts a new prompt read, and the cancelled turn also costs the conversation-cache slot (#1620) — the reporter counted ~740k re-read tokens in one day.

## Change

The wait is now a non-blocking acquire + a `ping` every `FIFO_PING_S` — the same 10 s heartbeat the stream already sends while the engine is quiet, so every consumer (OpenAI SSE, Anthropic, Responses) turns it into keep-alive bytes unchanged. `STRATA_FIFO_PING_S` overrides the interval.

Side benefit: a disconnect while queued is now noticed at the next ping's failed write — the silent block could wait forever holding the queue position.

Unchanged: `--batch` requests never held the fifo; the turn's release still runs through the lock's own `__exit__` (the `StatusHandover` test's armed lock exercises exactly that).

## Test plan

- New `QueuedHeartbeat` test: holds the fifo, asserts the request yields `ping`s, releases, asserts it runs to `done` and the lock is freed.
- `StatusHandover` (the fifo's bespoke lock double) passes — caught and fixed an earlier version that used `acquire(timeout=)`, which that double does not support.
- Full `serve.test_server`: 272 tests; 3 pre-existing errors are `ModuleNotFoundError: regex` in the real-tokenizer tests (env dependency, unrelated).
- Not tested: a real client with an idle timeout — the mechanism is the same keep-alive bytes the engine-quiet path already sends.

En el sitio

Enlaces a install, modelos, releases.