Pull requests / #2057
#2057 serve: an opt-in "stream_all_tool_args" sends a tool call's non-string argument in pieces as the model writes it (#1666)
open · @psqq · 0 comentarios · En GitHub
Server & APIAMD / HIPModels & quantsDocumentationWindows
Descripción
Resolves #1666 (the keep-alive that already covers the idle timeout is in main as 14b9dd2e; this is the
other half: the argument itself stops being silence)
## Summary
A tool-call parameter whose declared type is not a string - an array, an object, a number - is held until
`</parameter>` and goes out as one `tool_args` piece, so an app that shows the arguments as they arrive sits on
`{"edits":` for minutes and then shows the finished call at once. `"stream_all_tool_args": true` in
`strata-<model>.json` sends that text as it is written, the way a string parameter already does. Off by default.
## What changed
- `serve/frontend.py`: `OutputParser` gains `stream_all_args`. In the `raw` state the value's JSON text passes
through as it arrives, holding back a possible partial `</parameter>` (the same `_hold` the `str` state uses) and a
candidate end whose follower has not arrived yet, so #210 still holds: a `</parameter>` inside the value stays part
of the value. When the parameter ends, only what has not gone out yet is sent. `_close_scan` closes a call cut
inside such a value with the rest of that text, not with a JSON string of it.
- `serve/server.py`: the config key `stream_all_tool_args` (must be true or false) reaches the parser.
- `serve/runconfig.py`: the key is editable from the About tab's Model settings card.
- `docs/DETAILS.md`: a bullet with the measurements; the streaming bullet now says a non-string parameter waits for
its end unless the switch is on.
- `serve/test_tool_args_stream.py`: 8 cases.
What it leaves alone: without the key the stream is byte-identical to before; string parameters are untouched; the
final `tool_call` of a call that was streamed is still not sent again, so the pieces a client joins are the
arguments; the value is still parsed and checked at the end, so a broken argument is still the error it was. With it
on, what the model wrote is what goes out, so the whitespace is no longer normalized.
## Extra Notes
Parser, an `edit` call whose `edits` is an array of two objects (217 characters), fed at a token width of 8: 15
argument pieces with one of 217 without the switch, 44 pieces with none longer than 9 with it. End to end over
`/v1/chat/completions` with `MockEngine` (one token per step): 17 argument deltas, largest 247 characters, without
the switch; 263 deltas, largest 8, with it; `finish_reason: tool_calls` and the joined pieces equal to the reported
arguments in both cases. On a real agent turn (Qwen3.8 Flash-Next IQ3_S, an `edit` call with two edits): 27 argument
deltas, 26 of them carrying 78 characters in all and one carrying 3,145 - 97% of the arguments in a single piece.
Checked with `python -m unittest serve.test_tool_args_stream serve.test_parser_track serve.test_reasoning_tools
serve.test_reasoning_rescue serve.test_frontend serve.test_runconfig serve.test_structured` (87 tests, OK) and
`serve.test_server` (291 tests; the only errors are the three that import the `regex` module, the same on an
unmodified tree). Not tested: Windows and AMD paths beyond their suites.
*🤖 Fully generated by an LLM (qwen3.8-flash-next-iq3_s)*En el sitio
Enlaces a install, modelos, releases.