Pull requests / #1673
#1673 serve: heartbeat a request that waits for the turn (#1619)
open · @nekomario28 · 0 Kommentare · Auf GitHub
Beschreibung
Fixes #1619. With the default single request turn a queued request received **no bytes at all**: the fifo was taken with a blocking acquire before the response's first yield. A client with a stream idle timeout aborts a healthy queued request, each retry starts a new prompt read, and the cancelled turn also costs the conversation-cache slot (#1620) — the reporter counted ~740k re-read tokens in one day. ## Change The wait is now a non-blocking acquire + a `ping` every `FIFO_PING_S` — the same 10 s heartbeat the stream already sends while the engine is quiet, so every consumer (OpenAI SSE, Anthropic, Responses) turns it into keep-alive bytes unchanged. `STRATA_FIFO_PING_S` overrides the interval. Side benefit: a disconnect while queued is now noticed at the next ping's failed write — the silent block could wait forever holding the queue position. Unchanged: `--batch` requests never held the fifo; the turn's release still runs through the lock's own `__exit__` (the `StatusHandover` test's armed lock exercises exactly that). ## Test plan - New `QueuedHeartbeat` test: holds the fifo, asserts the request yields `ping`s, releases, asserts it runs to `done` and the lock is freed. - `StatusHandover` (the fifo's bespoke lock double) passes — caught and fixed an earlier version that used `acquire(timeout=)`, which that double does not support. - Full `serve.test_server`: 272 tests; 3 pre-existing errors are `ModuleNotFoundError: regex` in the real-tokenizer tests (env dependency, unrelated). - Not tested: a real client with an idle timeout — the mechanism is the same keep-alive bytes the engine-quiet path already sends.
Mehr auf der Site
Links zu Install, Modellen, Releases.