#5839 [Engine] 0.2.13 quoted EOS still ends agent turns with reasoning-only stop, with opt-in recovery results
open · @RAFOLIE · 2 Kommentare · Auf GitHub
Setup & installServer & APINVIDIA / CUDAModels & quantsWindows
Beschreibung
### Area
[Engine] (Python serving layer and OpenAI/Anthropic APIs)
### Desktop app, Terminal or source?
Terminal (the `strata` command), released Windows package. The attached recovery changes are local Python serving-layer changes.
### Reproduction status
Reproduced locally on the packaged serving code before applying the local patch below.
### Strata version
Strata Terminal 0.2.13, Control 0.1.59, CUDA engine 0.1.59. The Terminal package records commit `d2f096475817d4666c364c820b0ca830f751cfc5`.
### Graphics card, memory and system
NVIDIA GeForce RTX 4090 24 GB, 64 GB system RAM, Windows 11 x64.
### Model, size and settings
Qwen3.8-Flash-Next, Unsloth UD-IQ3_XXS native GGUF, 262144 context, int8 KV, vision configured on, one inference request at a time.
The original configuration used DFlash2 strata1, `spec=4`, draft confidence 0.7, and a 2048 MiB VRAM reserve. The final recovery validation used drafting/MTP off, `spec=2` for the native IQ layout, suffix drafting and lookup off, and a 4096 MiB VRAM reserve. Prediction offers remained zero during that final run. These configuration changes are included here so the before/after evidence can be interpreted accurately. A same-settings A/B of the entire long-running workload remains unrun.
### What happened
A long tool-using agent session can finish a turn with partial `reasoning_content`, empty `content`, no tool calls, and `finish_reason="stop"`. The next tool step then never arrives. This happened with DeepSeek Harness and ZCode CLI, and a standalone request directly to Strata reproduced the empty-reply shape.
The smaller synthetic prompt below produced 3 empty replies in 6 attempts on the original serving code. The request allowed 2048 completion tokens. The three empty replies used 736, 217, and 346 completion tokens respectively. Each had the protocol completion marker and `finish_reason="stop"`.
A debug capture showed actual EOS id 248046 while the model was quoting a control marker in its reasoning. In the original `Service.run()`, the `t in self.stop_ids` path ended the pass with `stop`. The quote then remained unfinished and the client received only reasoning. Quoted or multiline `</think>` examples also exposed premature reasoning closure in `OutputParser`.
The desired behavior for an opt-in recovery mode is to preserve the quoted marker text and allow the pending answer or tool call to finish, within the request's existing output budget.
### Related discussions
This is fresh 0.2.13 / 0.1.59 reproduction and local recovery validation for the behavior discussed in #2236 and #1178, which were closed during tracker housekeeping. The maintainer discussion in #2236 proposed measuring an opt-in continuation. #1053 covers the existing `reasoning_close_retry` option, and #4077 covers related reasoning-delimiter handling. The local patch here keeps its additional behavior opt-in.
### Smallest steps or command that reproduce it
Start the affected model with the released serving code and thinking enabled. Set `STRATA_BASE_URL` to the local Strata server URL, then run this standard-library Python script. The request payload and streaming mode match the synthetic reproduction. Results vary between attempts.
```python
import json
import os
import urllib.request
base = os.environ["STRATA_BASE_URL"].rstrip("/")
prompt = "请解释字符串 `<|im_end|>`、`<|endoftext|>`、`</think>` 在本地模型输出和工具调用解析中的区别,给出它们的原样拼写和一个 JSON 示例。"
for attempt in range(1, 7):
body = {
"model": "local",
"max_completion_tokens": 2048,
"stream": True,
"stream_options": {"include_usage": True},
"messages": [{"role": "user", "content": prompt}],
}
request = urllib.request.Request(
base + "/v1/chat/completions",
data=json.dumps(body, ensure_ascii=False).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
content = ""
reasoning = ""
tool_indices = set()
finish = None
usage = {}
done = False
with urllib.request.urlopen(request, timeout=900) as response:
for line in response:
if not line.startswith(b"data: "):
continue
data = line[6:].strip()
if data == b"[DONE]":
done = True
continue
event = json.loads(data)
if event.get("error"):
raise RuntimeError(event["error"])
usage = event.get("usage") or usage
for choice in event.get("choices", []):
finish = choice.get("finish_reason") or finish
delta = choice.get("delta", {})
content += delta.get("content") or ""
reasoning += delta.get("reasoning_content") or ""
tool_indices.update(call["index"] for call in delta.get("tool_calls", []))
print(json.dumps({
"attempt": attempt,
"finish_reason": finish,
"done": done,
"content_chars": len(content),
"reasoning_chars": len(reasoning),
"tool_calls": len(tool_indices),
"usage": usage,
}))
```
The synthetic prompt asks for an explanation of the literal spellings of the three markers and a JSON example.
### Logs
The following fields are extracted from the three failing synthetic requests. Only numeric response metadata is included.
| Attempt | Prompt tokens | Completion tokens | Content characters | Reasoning characters | Tool calls | Finish | Protocol done |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2 | 98 | 736 | 0 | 2095 | 0 | stop | true |
| 3 | 98 | 217 | 0 | 441 | 0 | stop | true |
| 6 | 98 | 346 | 0 | 756 | 0 | stop | true |
### Local opt-in recovery and checks
The patch below adds `quoted_eos_recovery`, default off, alongside the existing `reasoning_close_retry`.
- In open code/quotation contexts, encode the emitted EOS spelling as ordinary text tokens and continue the same output.
- Track fences, inline backticks, and unescaped double quotes to preserve quoted/multiline reasoning delimiters.
- Extend the one-time empty-answer reasoning closure to the held/just-closed reasoning delimiter cases under this opt-in.
- Charge inserted tokens against output and reasoning budgets, preserve explicit stop strings and cancellation, and retain exactly-once delivery of subsequent tool calls.
Quoted-context detection is heuristic. The end-to-end validation here covers this Qwen model, the stated Windows/CUDA setup, and the two APIs. Broader model/template compatibility remains unverified.
Final continuous validation with both recovery options enabled completed the following.
| Check | Observed result |
| --- | --- |
| OpenAI-compatible streaming | 160 actual local tool-worker steps, 161 responses, 69m39s, 0 empty replies, final completion acknowledged |
| Anthropic-compatible streaming | 80 actual local tool-worker steps, 81 responses, 14m22s, 0 empty replies, final completion acknowledged |
| DeepSeek Harness client | 60 worker steps, 0 tool errors, final completion acknowledged |
| ZCode CLI client | 60 worker steps, 63 model requests, all on first attempt using the intended local model, 0 tool errors |
| Long tool waits | Three 310-second waits across the two API loops, automatic continuation after each, plus 65-second waits |
| Reconstructed original failing request | 10 replays, 49492 prompt tokens and 82 declared tools, valid tool calls in all 10 |
| Literal-marker JSON copying | 20 requests with exact parsed JSON values and preserved marker characters, all final completion markers present |
| Long conversation then marker copying | 163686 prompt tokens, 21 repeated end markers, exact JSON values, normal completion |
| Large-context follow-up | 243697 prompt tokens after adding repeated padding to the completed history, exact JSON values, normal completion |
| Unit/regression suite | 491 run, 488 passed, 3 Windows signal-dependent tests skipped, including 17 added regression cases |
The four tool loops total 360 executed worker steps. Tool sequence, returned values, and SHA256 results were independently checked. The 243697-token follow-up was a separate padded request. The naturally growing OpenAI tool conversation reached 163387 input tokens. The saved reconstructed-request replays checked returned tool calls, and the separate controlled workers supplied the actual execution checks.
The final continuous server-log segment had one startup-ready record, 36 quoted-EOS continuation events, and no server-error headers. A later controlled restart with debug output disabled passed tool-call/final-answer checks on both APIs and another 21-marker JSON copy.
The original 2048-token diagnostic prompt was also rerun 6 times after the local changes. It produced 0 empty replies, 2 normal stops, and 4 `length` results at exactly 2048 completion tokens. Those four runs are output-budget boundary results. The completion and JSON-integrity results in the table above came from their separate workloads.
The regression command actually run, with installation-specific environment paths omitted, was
```text
python -m unittest serve.test_server serve.test_frontend serve.test_reasoning_rescue serve.test_reasoning_tools serve.test_stop_holdback serve.test_tool_call_unclosed serve.test_quoted_eos_recovery serve.test_parser_track serve.test_parser_hold
```
The release archive omitted two developer test helpers, which were supplied from the same recorded upstream commit for that run. The three skips were `StartCrashReported` signal tests on Windows.
### Focused local patch
The following diff contains the two serving files and the 17-case regression module. All paths are repository-relative. AI assistance was used to prepare the local changes, run validation, and draft this report.
<details>
<summary>Serving-layer diff and regression tests</summary>
```diff
diff --git a/serve/frontend.py b/serve/frontend.py
--- a/serve/frontend.py
+++ b/serve/frontend.py
@@ -1179,6 +1179,7 @@
self.state = "reasoning" if thinking else "content"
self.buf = ""
self.lead = False
+ self.line_think_end = False # opt-in: a closing reasoning tag is followed by a line break
self.schemas = {t.get("name"): t for t in tools or []}
# #4663: the request defers tools (Anthropic's tool search, see anthropic_tools): the client holds back more tools
# than it declares, so a call for an undeclared name is a call too - Claude Code runs it, where text ended the turn.
@@ -1237,7 +1238,7 @@
self.after_think = False # #4077: the reasoning ended at a bare `</think>` (a second one in the answer is a stray)
self.stray_think = 0 # ... and the strays of that kind dropped from the answer
self.refused = 0 # declared calls kept as reasoning text (quoted, or the turn was cut)
- self.fence, self.line, self.ticks = "", "", 0
+ self.fence, self.line, self.ticks, self.quotes = "", "", 0, 0
self._prev_blank = self._indented = False # #4703: the line before was blank / this is inside an indented (4-space) code block
# recover (on in the server since 0.2.8, off for a bare OutputParser; the config's "tool_call_recovery": false turns it off): a declared tool's call written next to the template's
# form is the call - `<parameter=NAME>` as the opener, JSON inside <tool_call>, a bare <function=NAME> at the
@@ -1254,6 +1255,20 @@
"""Nothing is held back: a clean point to stop the thinking at (a parser that rewrites the model's text, as
serve/dsml.py's does, also counts what it holds)."""
return not self.buf and not self._held
+
+ def quoting_control_token(self) -> bool:
+ """An end token in an open code example or JSON string is being quoted as text."""
+ if self.state not in ("reasoning", "content"):
+ return False
+ line = self.line + self.buf
+ if self.fence:
+ closer =Mehr auf der Site
Links zu Install, Modellen, Releases.