Pull requests / #1196

#1196 serve: share the thinking budget with other apps, like max_tokens and…

closed · @elanonimo832 · 0 commentaires · Sur GitHub

Server & API

Description

… the effort

The Chat settings a client posts to /settings reach every client through `shared`, and `reasoning_effort` and `max_tokens` are among them. The thinking budget (#123) was not, so a client that sets none of its own kept thinking to the ceiling while the Chat settings asked for less.

Two changes make it work like the other shared keys:

- `reasoning_budget_tokens` joins SHARED_KEYS, validated as a whole number of tokens. 0 keeps its documented meaning ("no budget"), so it is allowed there where seed and max_tokens require > 0.
- `Service.run` reads the budget *after* `sampling_defaults` and `shared` are merged into `sampling`. It read it before the merge, so no shared key could ever reach it. The request's own value still wins (an explicit `reasoning_budget_tokens: 0` keeps turning a shared or configured one off), and the config's `reasoning_budget_tokens` stays the fallback.

Tests: `test_a_shared_budget_reaches_a_request_that_sets_none` (fails with "1 != 2" on the engine prompts before the change) and `test_a_shared_budget_must_be_a_whole_number_of_tokens`, next to the existing budget tests. ThinkingBudget 11/11 pass, SharedSettings 6/6 pass.

Rebased onto the new `main` after the history cleanup. Replaces #1035.

Sur le site

Liens install, modèles, releases.