Pull requests / #2397
#2397 serve: allow JSON response_format with tools on /v1/chat/completions
open · @Xantier · 0 comentarios · En GitHub
BenchmarksServer & APIModels & quantsDocumentation
Descripción
Issue: none. Follows #782, and picks up the structured-output part of #763, which closed when main was force-pushed. ## Summary POST /v1/chat/completions returned 400 structured response_format with tools/MCP is not supported for any request that sent both tools and a JSON response_format. #782 made /v1/responses accept this combination. This PR applies the same approach to the chat route: a turn that calls a tool passes through unchanged, and the turn that ends in text is validated against the schema as before. Measured with tool-eval-bench 2.8.1.dev1 (69 scenarios, seed 42, 3 trials each) on an RX 9070 16 GB, Ryzen 5 3600 and 64GB of DDR4-2400, running Qwen3.8-Flash-Next IQ3_S on engine 0.1.43.1 with reasoning_effort: low: | Build | Mean points (3 trials) | TC-65, TC-66, TC-67, TC-69 | |---|---|---| | main | 128.3 / 138 | partial in every trial (400 on the turn after the tool result) | | this PR | 131.3 / 138 | pass in every trial, no 400s | ## What changed - serve/server.py, _openai: prepare_format gets with_tools=bool(tools), so the system directive says the schema is for the final answer only. This is the directive #782 added. The 400 now applies only to strata_mcp. - serve/server.py, structured_chunks: the first chunk with tool_calls releases what was held and streams the rest unchecked. A turn without a tool call is buffered and validated exactly as before. - serve/test_structured.py: the assertion that expected 400 for tools now covers MCP only. New tests cover a tool call passing through (streamed and non-streamed), a valid final answer with tools, and a final answer that breaks the schema (502 structured_output_failed). - docs/DETAILS.md: describes the tools behaviour in place of "refused explicitly". Left alone: requests without response_format, and JSON formats without tools, take the same path as before, and the existing tests pass unchanged. MCP with a JSON format is still refused, because run_with_mcp runs tool rounds inside one request, so there is no single final turn to validate. ## Extra Notes - Tests on this branch (v0.1.44.1 base, Python 3.12, jsonschema 4.26.0): serve.test_structured 13, serve.test_responses 40 and serve.test_server 314, all OK. - This does not conflict with #1328, which leaves the chat route's 400 unchanged. If there is a reason to keep the chat route stricter than /v1/responses, I'd like to hear it.
En el sitio
Enlaces a install, modelos, releases.