Pull requests / #1788
#1788 serve: preserve tool prefixes for tool_choice=none
open · @llj098 · 0 comments · View on GitHub
BenchmarksServer & APIModels & quants
Description
## Summary Keep tool declarations in the Chat Completions prompt when `tool_choice` is `"none"`, while independently disabling structured tool-call output and the MCP tool-execution loop. A client can keep its conversation and tools unchanged, append a summarization/compaction instruction, and set `tool_choice: "none"`. Currently `_openai()` replaces `tools` with `None`. Because the model's template puts tool declarations near the beginning, this changes the leading prompt tokens and prevents an otherwise matching conversation checkpoint from being reused. A long context is not necessary to reproduce this input-prefix mismatch. Related: #1222 addressed Codex/Responses compaction. This PR targets `/v1/chat/completions`; it does not change the Responses compaction mechanism or require client/session metadata. ## What changed - Retain prompt tool declarations for `tool_choice: "none"`. - Add keyword-only `parse_tools=True` to `OutputParser` and `Service.run`. Chat Completions sets it to false for `none`, including disabling recovery of calls inside reasoning. Preserve the existing `tool_call_recovery` option for other requests. - Do not enter `run_with_mcp` under `none`; MCP declarations can remain in the prompt without executing model-requested tools. - Preserve acceptance of `response_format` with tool declarations when `tool_choice` is `none`; the existing structured-output/MCP restriction remains. - Add regression coverage for equal absent/auto/none prompt tokens, streamed/non-streamed responses, thinking on/off, split and truncated tool markup, recovery on/off, MCP non-execution, and JSON response format. If the model writes tool-shaped text under `none`, it is returned as ordinary reasoning/content, **not** `tool_calls` or a `tool_calls` finish reason. This is an output-parsing/execution restriction, not a new grammar or sampling constraint. Normal `auto`/absent/required/named behavior, Responses and Anthropic routing, the native engine, and KV formats/policies are left alone. The branch is based directly on upstream `fb58e0dbc8399662c0e47c76578c6e878b14f6cf` and contains only this change, in four Python files. ## Validation ### CPU/mock regressions ```sh python -m unittest \ serve.test_frontend serve.test_server serve.test_reasoning_tools \ serve.test_structured serve.test_responses serve.test_mcp ``` **372 tests passed** on the PR worktree. No GPU backend files changed. ### Short-context prefix reproduction Compared unmodified upstream `fb58e0d` with this patch using the real model tokenizer/chat template, actual isolated HTTP handlers, and `MockEngine` generation. The fixture has two tool definitions and 128 short synthetic records, followed by an assistant reply and a summarization request with `tool_choice: "none"`. The context limit is 4096 tokens. | Measurement | Before | After | |---|---:|---:| | Ordinary request input tokens | 3196 | 3196 | | Appended compaction request input tokens | 2948 | 3237 | | Common prefix with ordinary request | 41 | 3196 | | Native prefix predicate's matching length | 0 | 3191 | The ordinary request's token sequence is identical before/after. For the final row, a **synthetic checkpoint token sequence** ending before the first assistant turn (3191 tokens) was supplied to the **unmodified upstream C++ `conversation_prefix`** function. This checks HTTP rendering and native **token matching only**: no actual KV/GDN snapshot was captured or restored, and no GPU inference or cache-hit/performance benchmark was performed. It removes this specific source of prefix invalidation; it does not guarantee hits when other prompt content or checkpoint boundaries change.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.