Pull requests / #2397

#2397 serve: allow JSON response_format with tools on /v1/chat/completions

open · @Xantier · 0 commentaires · Sur GitHub

BenchmarksServer & APIModels & quantsDocumentation

Description

Issue: none. Follows #782, and picks up the structured-output part of #763, which closed when main was force-pushed.

## Summary

POST /v1/chat/completions returned 400 structured response_format with tools/MCP is not supported for any request that sent both tools and a JSON response_format. #782 made /v1/responses accept this combination. This PR applies the same approach to the chat route: a turn that calls a tool passes through unchanged, and the turn that ends in text is validated against the schema as before.

Measured with tool-eval-bench 2.8.1.dev1 (69 scenarios, seed 42, 3 trials each) on an RX 9070 16 GB, Ryzen 5 3600 and 64GB of DDR4-2400, running Qwen3.8-Flash-Next IQ3_S on engine 0.1.43.1 with reasoning_effort: low:

| Build | Mean points (3 trials) | TC-65, TC-66, TC-67, TC-69 |
|---|---|---|
| main | 128.3 / 138 | partial in every trial (400 on the turn after the tool result) |
| this PR | 131.3 / 138 | pass in every trial, no 400s |

## What changed

- serve/server.py, _openai: prepare_format gets with_tools=bool(tools), so the system directive says the schema is for the final answer only. This is the directive #782 added. The 400 now applies only to strata_mcp.
- serve/server.py, structured_chunks: the first chunk with tool_calls releases what was held and streams the rest unchecked. A turn without a tool call is buffered and validated exactly as before.
- serve/test_structured.py: the assertion that expected 400 for tools now covers MCP only. New tests cover a tool call passing through (streamed and non-streamed), a valid final answer with tools, and a final answer that breaks the schema (502 structured_output_failed).
- docs/DETAILS.md: describes the tools behaviour in place of "refused explicitly".

Left alone: requests without response_format, and JSON formats without tools, take the same path as before, and the existing tests pass unchanged. MCP with a JSON format is still refused, because run_with_mcp runs tool rounds inside one request, so there is no single final turn to validate.

## Extra Notes

- Tests on this branch (v0.1.44.1 base, Python 3.12, jsonschema 4.26.0): serve.test_structured 13, serve.test_responses 40 and serve.test_server 314, all OK.
- This does not conflict with #1328, which leaves the chat route's 400 unchanged. If there is a reason to keep the chat route stricter than /v1/responses, I'd like to hear it.

Sur le site

Liens install, modèles, releases.