Issues / #2236
#2236 Bug: engine truncates the answer when generated text contains the literal EOS token string
open · @tspine · 0 comments · View on GitHub
BenchmarksServer & APIModels & quants
Description
Bug: engine truncates the answer when the generated text contains the literal EOS token string
Summary
When the model's generated output contains the literal string for the EOS marker (the <|im_end|> token), the engine immediately stops generation. The result is finish_reason: "stop" with an empty or truncated content, while the reasoning phase rendered normally. From the client's point of view it looks like "the model thought, then gave no answer."
This is a different string from the </tool_call> literal fixed in v0.1.43.1 (#2026), so upgrading to 0.1.43.1 does not fix this one.
Environment
- Engine: strata 0.1.43 (server.py from the same tree)
- OS: Ubuntu 22.04.5 LTS, kernel 5.15.0-185-generic
- GPU: 2× NVIDIA L20 48G, dual-GPU layer split, no batch
- Model: Qwen3.8-Flash-Next UD-Q4_K_XL GGUF, --spec 4, MTP drafts, --max-context 262144
- Clients: OpenAI-compatible /v1/chat/completions with tools + reasoning_effort=medium, streaming on
Reproduction (minimal, deterministic)
Ask the model to reproduce the literal EOS string, e.g. (OpenAI chat, non-stream):
messages = [{"role": "user", "content": "Please output the literal EOS token string verbatim, then explain it in three sentences."}]
max_tokens = 2000
Observed with STRATA_DEBUG=1, the raw: dump shows generation stops exactly at the point where the string is emitted:
[strata] raw: 'The user wants me to:\n1. Output the string "<|im_end|>'
[strata] done: 14 tokens in 1 s (135.0 tok/s) (stop, cancel=False)
The content field comes back empty (finish_reason=stop), because the answer was cut before it finished.
Even mentioning the token as an example triggers it (the second minimal case):
messages = [{"role": "user", "content": "Explain what an end-of-text token is, in three sentences."}]
[strata] raw: '用户要求我解释"end of text token"的作用...\n\nEnd of text token(如 </s>、[EOS]、<|im_end|>'
[strata] done: 48 tokens in 1 s (108.9 tok/s) (stop, cancel=False)
Expected
The literal EOS string inside generated text should be treated as ordinary text (or escaped), not as a generation stop signal. Only the actual EOS token id should stop generation.
Why this matters / how it surfaced in practice
This bit us in a real workflow: an agent session (with tools + long history) happened to discuss Strata release notes, whose content quotes these literal token strings. The model then quoted them in its answer and killed its own response, producing "thought but no answer" across multiple clients. Because the poisoned context gets persisted (e.g. memory/context injection), it leaked into new sessions until the offending history was removed.
Note on v0.1.43.1
Please confirm whether this EOS-literal case is covered by any upcoming fix. #2026 fixed the </tool_call> literal, but the <|im_end|> literal appears to still be unhandled in 0.1.43.1.Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.