Pull requests / #1035
#1035 serve: share the thinking budget with other apps, like max_tokens and…
closed · @elanonimo832 · 0 comentários · No GitHub
Descrição
… the effort
The Chat settings a client posts to /settings reach every client through `shared`, and `reasoning_effort` and `max_tokens` are among them. The thinking budget (#123) was not, so a client that sets none of its own kept thinking to the ceiling while the Chat settings asked for less.
Two changes make it work like the other shared keys:
- `reasoning_budget_tokens` joins SHARED_KEYS, validated as a whole number of tokens. 0 keeps its documented meaning ("no budget"), so it is allowed there where seed and max_tokens require > 0.
- `Service.run` reads the budget *after* `sampling_defaults` and `shared` are merged into `sampling`. It read it before the merge, so no shared key could ever reach it. The request's own value still wins (an explicit `reasoning_budget_tokens: 0` keeps turning a shared or configured one off), and the config's `reasoning_budget_tokens` stays the fallback.
Tests: `test_a_shared_budget_reaches_a_request_that_sets_none` (fails with "1 != 2" on the engine prompts before the change) and `test_a_shared_budget_must_be_a_whole_number_of_tokens`, next to the existing budget tests. ThinkingBudget 11/11 pass, SharedSettings 6/6 pass.No site
Links install, modelos, releases.