#6204 [Engine] opt-in merging of consecutive text/thinking deltas on /v1/messages

open · @matunoki · 0 コメント · GitHub で見る

Server & APIModels & quantsDocumentationWindowsLinux

本文

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

When I use the Claude Code VS Code extension with Strata over Remote-SSH, the /v1/messages stream arrives as one event per token, and the extension's panel falls behind and freezes. I suspect the number of events is the cause. After I put a proxy in front of Strata that merges the events, the freezing stopped in my environment. Before that, it froze almost every time, both when connected directly to Strata (one test) and through an earlier version of my proxy that did not merge events. This is a before/after observation, not a controlled comparison.

So, **could you add an opt-in setting that merges consecutive text and thinking deltas of the same content block into fewer events?** I am asking for the setting to be off by default; when it is off, Strata would behave exactly as it does today. This is not a request to change the default behavior.

I have also reported the problem on the extension side to Claude Code (see the details below). This setting would be separate from that report: it would let Strata users get the benefit without running a proxy. Before any implementation work, I would like to ask whether this direction fits the project.

**The behavior I am asking for**
1. Could you add one new setting? The name and form are up to you, whatever fits Strata best. Examples:
   - A command-line flag `--stream-interval N` (merge every N tokens; the default is 1, which is the same as today). vLLM (<https://docs.vllm.ai/en/stable/configuration/engine_args/>) and SGLang (<https://docs.sglang.io/docs/advanced_features/server_arguments>) have a setting with this name (both count tokens, and the default is 1).
   - Or a time-based setting (for example a config key or an environment variable that takes milliseconds; 0 means off). This is the form I tried (200 ms, up to 256 characters). I note one caveat about the time-based form in the details below.
2. With the setting off (the default), the output stays exactly as it is today.
3. With the setting on, consecutive text_delta / thinking_delta events of the same block are merged into one event. The concatenated text would be the same as today. I would expect the first delta of a block to be sent immediately, and anything held back to be flushed before the block ends, before the message ends (message_stop), and before an error is reported. All other events would keep their content and order.
4. **I think tool arguments (input_json_delta) do not need to be included.** In my measurements (a fake server, with the settings the extension uses in a remote window), only thinking increased the number of messages the CLI emits; tool arguments had no effect. So they are not needed for this problem. This also avoids overlapping with #5628 (a PR that streams tool arguments in finer-grained deltas).

<details>
<summary>Details (environment, measurements, code locations)</summary>

I wrote this part expecting that the maintainers' AI agents may read it too.

**What I saw in my environment**
- Engine 0.1.60.1 (build_info from /props), model Qwen3.8-Flash-Next IQ3_XXS. Client: the Claude Code VS Code extension 2.1.326. The VS Code window runs on Linux Mint 22 and connects through Remote-SSH (RTT about 130 ms) to the extension host on Windows 11, where the extension and its CLI run.
- The first few turns responded normally. After that, the panel stayed on "working" and stopped responding. When I reloaded the client window, the work on the host had finished and the history was visible. This happened almost every time.
- In the same window over the same SSH connection, this did not happen when I switched to a Claude model (official API).

**Strata's current behavior**

From reading the code, it looks like /v1/messages creates one text_delta or thinking_delta event for each parser event and flushes after each one (`anthropic_events` and the HTTP handler's `_anthropic` in `serve/server.py`, v0.2.15). I found no setting for this granularity. docs/DETAILS.md (v0.2.15) also says "everything arrives as it is made", so I understand this is intended behavior, not a bug.

I have been recording the streamed responses through my own local routing proxy (it routes requests to models and logs each response). Here is a summary of those records.

| Item | Value |
|---|---|
| Recorded responses | 14,126 responses (Oct 23 to Nov 17) |
| Events per token | median 0.91 (about 90% of responses had at least 0.74), so close to one event per token |
| Responses with 1,000 or more events | 4,097 responses |
| Generation speed | about 63 events/s (median), about 153 events/s (peak) |

Share by event type (an estimate; 16,071 responses from the same records, counted separately and including Nov 18, so the total differs):

| Type | Share |
|---|---|
| thinking | about 64% |
| text | about 8% |
| tool arguments | about 28% |

The records do not keep a per-block event count, so this is an estimate: I split each response's total event count in proportion to the character count of each block.

**What I measured on the Claude Code extension side**

The Claude Code extension runs a CLI (`claude`) in the background and sends each message the CLI emits to the webview one by one. So I sent 600 events from a fake server, one delta per event (60 events/s), and counted the messages the CLI emitted. I used the settings the extension uses in a remote window (partial messages disabled), with one run per condition.

| Delta sent (600 events) | CLI messages (as is) | When merged every 0.2 s |
|---|---|---|
| thinking | 605 | 56 |
| text | 4 | 4 |
| tool arguments | 6 | 6 |

Only thinking increased the number of messages.

**How to check this with Strata (message count only; Remote-SSH is not needed)**

Connect the Claude Code CLI directly to Strata and count the messages it emits, grouped by type. This check only shows that fewer events from Strata lead to fewer CLI messages. It does not show that the extension stops freezing.

```
ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_API_KEY=dummy ANTHROPIC_MODEL=<your Strata model name> \
  claude -p "Think step by step, then answer: what is 17 * 23?" \
  --bare --no-session-persistence --output-format stream-json --verbose > out.jsonl

python -c "import json,collections;c=collections.Counter((lambda m:m['type']+'/'+m.get('subtype',''))(json.loads(l)) for l in open('out.jsonl',encoding='utf-8'));print(*sorted(c.items()),sep='\n')"
```

The number of `system/thinking_tokens` messages is one per thinking event. In my environment (one run each), it was 37 when connected directly to Strata and 4 through the proxy that merges every 0.2 s. The longer the thinking, the larger the difference.

I also wrote about this in the Claude Code issues: anthropics/claude-code#110406 (my comment: <https://github.com/anthropics/claude-code/issues/110406#issuecomment-6974350389>) and anthropics/claude-code#86854 (almost the same problem; closed and then locked). I do not know whether the extension itself will be fixed.

**What I tried**
- I put a small proxy between Claude Code and Strata. It merges consecutive text / thinking / tool-argument deltas of the same block: at most one event per 0.2 s, or sooner once 256 characters are held. It sends the first delta immediately and leaves all other events unchanged. The number of events dropped to about 1/15 of the original count.
- I published a simplified version of this proxy as a single file (only aiohttp is needed): https://gist.github.com/matunoki/e67fef436a360167d16100038d271a06
- I used three sessions in parallel for about 3 hours and nothing froze. I did not measure timings.

**A point to keep in mind for the time-based form**

A time-based setting may need a way to flush what is held back when events stop arriving. From reading the v0.2.15 code, events seem to come out of a synchronous generator one by one, and there seems to be no timer. When nothing is sent, a ping goes out about every 10 seconds (`KEEPALIVE_S` = 10, plus the engine's heartbeat). So it looks possible to flush the held deltas when the next event or a ping arrives, but if generation pauses midway, the held text could be delayed by up to about 10 seconds (I expect this would not matter in the normal flow of tokens). A token-count setting would not have this concern. I only read the code; I have not implemented or tried it.

**Related, and what I have not confirmed**
- Related: #5628 (finer streaming of tool arguments, a different scope), #1951 and #4340 (keep-alive while nothing is sent; too few events, the opposite of this request), #6177 (a different Claude Code delay: permission-check requests that wait behind a long prompt read).
- Not confirmed:
  - The effect if this were implemented in Strata.
  - The cause-and-effect between many messages and the extension freezing. This is my inference from reading the extension's code. What I have is a before/after observation in my environment: it froze almost every time without the merging (connected directly, and through the earlier proxy version), and it did not freeze in three sessions over about 3 hours with the merging. I did not run a controlled comparison, such as switching the merging on and off in the same session. The reporter of anthropics/claude-code#86854 measured how the delay changes with the merging window.
  - Whether other engines show the symptom (the reporter of anthropics/claude-code#86854 describes the same symptom with a provider that emits one event per token).
  - How other clients behave.

</details>

### What do you do today instead?

I run a merging proxy in front of Strata. A simplified version is published as a single file (only aiohttp is needed): https://gist.github.com/matunoki/e67fef436a360167d16100038d271a06 . This workaround needs a separate process, so Strata's own settings are not enough.

---

**Testing**

If this setting is added, I would be happy to test it and report the results.

I wrote this in Japanese and translated it into English with the help of AI, so some of the wording may sound unnatural. Please let me know if anything is unclear.

Thank you for Strata. I use it every day and it is very useful to me.

関連リンク

インストール・モデル・リリースへのサイト内リンク。