Pull requests / #2039
#2039 serve: preserve literal </tool_call> in prompts
open · @CC-David-CC · 0 Kommentare · Auf GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Beschreibung
## Title serve: preserve literal `</tool_call>` in prompts Issue: Resolves https://github.com/Niko1221/Strata/issues/2026 ## Summary Asking the model to print `</tool_call>` verbatim can produce an empty final response. The tokenizer treats this spelling as a user-defined special token even inside ordinary message text. Protecting it as plain text in the prompt makes the reported reproduction return the requested string. ## What changed - Add `TOOL_LITERAL_TAGS` with one entry, `</tool_call>`, to the existing literal-span protection. - Protect occurrences in messages, assistant history/reasoning, tool results, definitions, and argument values while preserving template-generated tool-call closers as special tokens. - Add regressions for prompt token IDs, literal model output, all two-part streaming boundaries, real tool calls, explicit stop sequences, and streaming/non-streaming HTTP responses. This is a standalone fix against `main`; it does not depend on #1886. No new state machine, retry, configuration option, or engine change is required. Prompts without the target literal retain identical token IDs. ## Extra Notes ### Validation The issue's exact prompt was run through the service with a real IQ3_XXS model on Linux/CUDA (RTX PRO 6000 Blackwell). Ten interleaved before/after pairs used seeds 42-51, temperature 0.6, top_p 0.95, top_k 20, thinking enabled, and no tools. Both arms used the PR code; the baseline restored the original literal registry. Raw input/output token IDs were captured. | Result | Original registry | With fix | | --- | ---: | ---: | | Exactly `</tool_call>` | 0/10 | 10/10 | | Empty final response | 9/10 | 0/10 | | Incorrect `<\|im_start\|>` response | 1/10 | 0/10 | The `</aaa>` and `</mcp_tool>` controls each passed before and after. The reporter's IQ3_S/Windows/AMD configuration was not tested. 339 tests passed on the standalone PR branch (Windows, Python 3.13.15). Regression command (no model/GPU required): ```sh python -m unittest serve.test_literal_tool_tags serve.test_frontend serve.test_reasoning_tools serve.test_tool_name_space serve.test_server -q ``` Removing the new registry entry makes 20 prompt-protection subcases fail; restoring it passes them. Existing output parsing already preserves a standalone closer, so the production fix stays in prompt tokenization.
Mehr auf der Site
Links zu Install, Modellen, Releases.