Pull requests / #1678

#1678 serve: keep-alive while a tool call's held-back argument is generated (#1666)

open · @nekomario28 · 0 comments · View on GitHub

Server & API

Description

Fixes #1666. (The patch is the reporter's, applied to main; this adds the ping-cadence unit test the issue offered.)

With `stream_tools`, `OutputParser` holds an array/object argument until `</parameter>` — a large one (a measured 66 KB `edit_file` call) left the stream silent for minutes while tokens were generated, so a client's idle timeout ended a healthy request. Forge's is 120 s; the reporter hit it on a real agent turn.

## Change

`Service.run`'s token loop now tracks `last_out` (set on the heartbeat ping and whenever the parser emits events). When a token produces no event and nothing has gone out for `KEEPALIVE_S` (10 s), it yields the same `"ping"` the engine-quiet heartbeat sends — no new event type; clients already ignore SSE comments. Ordinary answers are untouched: a token that streams content keeps resetting `last_out`.

## Test plan

- New `HeldBackArgKeepalive` in `serve/test_server.py`: a fake engine whose generated `<tool_call>` is held whole by the frontend (the same silence a long array argument makes), over a patched clock — asserts a `ping` still goes out and the request reaches `done`; and a control that an ordinary answer emits no ping.
- Full `serve.test_server`: 273 tests, 3 errors all `ModuleNotFoundError: regex` — the missing optional dep in this environment, identical on a clean checkout; unrelated.
- Reporter's live reproduction: longest gap between SSE events 27.8 s → 10.0 s, the 7.6 KB array arriving intact; an 880 s Forge agent turn with a 66 KB argument kept alive.

Not tested here: a real 120 s idle-timeout client (the reporter measured that end); the keep-alive itself is covered above.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.