#6022 [Engine] A tool-calling model can hallucinate a tool name and leak raw tool-call markup into the reply

open · @sergey-zinchenko · 5 コメント · GitHub で見る

Server & API

本文

### Area

[Engine] (speed, kernels, models, the server and API)

### What do you want to do?

A model serving tool calls can invent a tool name that no declared tool has, and the raw call markup then leaks into the reply text. Observed live on Qwen3.8-Flash-Next (Unsloth `UD-Q4_K_XL`) behind the OpenAI-compatible server, driving three different tool-calling agents — **Claude Code, Hermes, and Cursor** — each declaring a large tool set (about four dozen in each case):

- the model wrote `<tool_call><function=NAME>` with a `NAME` no declared tool has (`TaskCreate`, `Delete`);
- the reply parser dropped the call, so `tool_calls` was empty and the raw `<tool_call>…</tool_call>` text landed in `message.content` with `finish_reason=stop`; the agent then had no call to run and stopped. This repeated across turns.

The same model also writes other spellings of a call, which the parser does not read and which leak the same way:

- the XML `<invoke name=".."><parameter name="..">` form (alone, inside `<tool_call>`, or inside `<function_calls>`);
- `<function=T>{json}</function>` — a JSON object as the body;
- `<function=T><parameter name="k">v</parameter>` — the name as an attribute.

The goal: a declared tool call ends as `tool_calls`, and no tool-call markup leaks into the reply text.

Proposed behavior:

- Inside a real `<tool_call>` envelope (past the reasoning block), constrain the sampler so that after `<function=` only tokens that continue a declared tool name can be sampled. Opt-in and per-request; a request without tools pays nothing.
- Teach the reply parser to read the other three spellings for declared tool names, so they become calls instead of text.

Trade-offs:

- Speculative decoding stays on: a multi-token verify window is sampled under a per-row mask, so a drafted name no tool declares is never accepted.
- The trigger is the longest common prefix of every `<function=NAME>` tokenization; if the names diverge at their very first token the constraint cannot arm and the request falls back to today's behavior.
- Only names declared on the request are constrained; deferred tools and raw passthrough keep today's behavior.

### What do you do today instead?

None — the call is dropped and the turn ends with no tool call; retrying the same step reproduces the same hallucinated name.

### Evidence

Screenshots of the leaked markup from Claude Code, Hermes, and Cursor will be attached here.

<img width="1280" height="771" alt="изображение (2)" src="https://github.com/user-attachments/assets/8dc18bca-ffcc-4430-8702-cf1e35e8bbe5" />
<img width="1280" height="542" alt="изображение (1)" src="https://github.com/user-attachments/assets/688c1426-ed11-4b4d-9df7-16b15cf1fa68" />
<img width="1280" height="174" alt="изображение" src="https://github.com/user-attachments/assets/5c0d458c-2ee6-4671-8bdb-00805aafea5e" />

関連リンク

インストール・モデル・リリースへのサイト内リンク。