Pull requests / #2039

#2039 serve: preserve literal </tool_call> in prompts

open · @CC-David-CC · 0 Kommentare · Auf GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

## Title
serve: preserve literal `</tool_call>` in prompts

Issue: Resolves https://github.com/Niko1221/Strata/issues/2026

## Summary

Asking the model to print `</tool_call>` verbatim can produce an empty final response. The tokenizer treats this spelling as a user-defined special token even inside ordinary message text. Protecting it as plain text in the prompt makes the reported reproduction return the requested string.

## What changed

- Add `TOOL_LITERAL_TAGS` with one entry, `</tool_call>`, to the existing literal-span protection.
- Protect occurrences in messages, assistant history/reasoning, tool results, definitions, and argument values while preserving template-generated tool-call closers as special tokens.
- Add regressions for prompt token IDs, literal model output, all two-part streaming boundaries, real tool calls, explicit stop sequences, and streaming/non-streaming HTTP responses.

This is a standalone fix against `main`; it does not depend on #1886. No new state machine, retry, configuration option, or engine change is required. Prompts without the target literal retain identical token IDs.

## Extra Notes

### Validation

The issue's exact prompt was run through the service with a real IQ3_XXS model on Linux/CUDA (RTX PRO 6000 Blackwell). Ten interleaved before/after pairs used seeds 42-51, temperature 0.6, top_p 0.95, top_k 20, thinking enabled, and no tools. Both arms used the PR code; the baseline restored the original literal registry. Raw input/output token IDs were captured.

| Result | Original registry | With fix |
| --- | ---: | ---: |
| Exactly `</tool_call>` | 0/10 | 10/10 |
| Empty final response | 9/10 | 0/10 |
| Incorrect `<\|im_start\|>` response | 1/10 | 0/10 |

The `</aaa>` and `</mcp_tool>` controls each passed before and after. The reporter's IQ3_S/Windows/AMD configuration was not tested.

339 tests passed on the standalone PR branch (Windows, Python 3.13.15). Regression command (no model/GPU required):

```sh
python -m unittest serve.test_literal_tool_tags serve.test_frontend serve.test_reasoning_tools serve.test_tool_name_space serve.test_server -q
```

Removing the new registry entry makes 20 prompt-protection subcases fail; restoring it passes them. Existing output parsing already preserves a standalone closer, so the production fix stays in prompt tokenization.

Mehr auf der Site

Links zu Install, Modellen, Releases.