Pull requests / #3340

#3340 [Other] serve: accept pi's three thinking-budget field names as aliases

open · @shizhx · 0 commentaires · Sur GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux

Description

Problem and result: pi's `compat.thinkingTokenBudgetField` only allows
`thinking_token_budget` (vLLM), `thinking_budget` (Qwen/DashScope/SGLang) and
`thinking_budget_tokens` (llama.cpp), but #123's thinking budget was only readable
from `reasoning_budget_tokens`, so a pi client's budget was silently ignored - the
model thought its full length instead of being wrapped up at N tokens. After this
change all three names work exactly like `reasoning_budget_tokens` on the three
chat routes. Related: issue #123 #984.

## Summary

```diff
_post (serve/server.py)
  req = json.loads(body)
  if path in chat routes:
    self.record = svc.begin_request(path, req)
+   if req.get("reasoning_budget_tokens") is None:
+     first of thinking_token_budget / thinking_budget / thinking_budget_tokens
+       -> req["reasoning_budget_tokens"]
  dispatch -> _responses / _openai / _anthropic
    svc.reasoning_budget(req)      # existing: reads the native name, 400 on a non-integer
```

One commit, 3 files, +49/-1 (`serve/server.py` alias copy, `serve/test_server.py`
4 tests, `docs/DETAILS.md` the budget bullet). A request that sends none of the
three names goes through the added lookups unmodified: no mutation, no config,
engine or response change, so the default path is unchanged.

## Evidence

- **Before:** on base e88ee39d the three names are silently ignored

  ```text
  # fake-engine suite: the new tests against the unpatched server
  FAIL: test_the_budget_alias_field_names
  AssertionError: 'Let me think step by step about two plus two. ...' !=
                  'Let me think step by\n\nI have thought about this long enough ...'
  FAIL: test_a_bad_alias_is_a_400
  AssertionError: 200 != 400 : ('thinking_token_budget', 1.5, ...)
  Ran 23 tests in 13.215s - FAILED (failures=2)
  ```
- **After:** e88ee39d + this commit

  ```text
  test_the_budget_alias_field_names               red -> green
  test_a_bad_alias_is_a_400                       red -> green
  test_the_request_s_own_budget_wins_over_the_aliases   green (precedence guard)
  test_the_alias_field_order_wins                 green (precedence guard)
  Ran 23 tests in 13.231s - OK                    # serve.test_server.ThinkingBudget
  Ran 885 tests in 377.483s - OK (skipped=14)     # full serve suite, CI's command
  ```

  live on :4000 (real IQ3_S model, 2x RTX 2080 Ti), prompt "What is 17*23? ...",
  budget 20:

  ```text
  no budget (baseline)  200 finish=stop reasoning_chars=183, no wrap-up
  native                200 finish=stop wrapped=True reasoning_chars=181 answer='17 × 23 = 391'
  thinking_token_budget 200 finish=stop wrapped=True reasoning_chars=134 answer='17 × 23\n...'
  thinking_budget       200 finish=stop wrapped=True reasoning_chars=178 answer='17 × 23:...'
  thinking_budget_tokens200 finish=stop wrapped=True reasoning_chars=134 answer='...'
  bad alias (1.5)       400 reasoning_budget_tokens=1.5: expected a whole number of tokens
  ```

  Each alias wraps the thinking at 20 (wrap-up marker, answer after, finish stop)
  exactly like the native field; the response shape is unchanged.

## Approach

The alias copy happens once, at the request entry (`_post`, after `begin_request`,
before dispatch), so every downstream read - `Service.reasoning_budget` on all
three routes, including `/v1/responses` - sees the native name and keeps the
existing whole-number validation (a bad alias is the same 400). Precedence is
deterministic: the request's own `reasoning_budget_tokens` wins over all aliases;
among the aliases the fixed order `thinking_token_budget` > `thinking_budget` >
`thinking_budget_tokens` does. The patch applies verbatim onto e88ee39d: `do_POST`'s
body moved to `_post` and the req parse now happens after `reserve_inference`, but
the insertion point and the budget's read path are unchanged.

## Validation

Revision e88ee39d + 2b336465, Python 3.13.5,
Linux x86-64, 2x RTX 2080 Ti, Qwen3.8-Flash-Next IQ3_S:

- Passed:
  - `.venv/bin/python -m unittest serve.test_server.ThinkingBudget -v` - red
    before (the alias-wrap and bad-alias-400 tests fail on base; the two
    precedence tests pass since the native field already wins when the aliases
    are ignored) / green after, 23 OK
  - `.venv/bin/python -m unittest discover -s serve -p 'test_*.py'` (CI's
    command): 885 OK (14 skipped)
  - live requests on :4000 (`strata serve --profile strata-iq3_s.json`, the
    patched tree): see Evidence
- Failed: none observed
- Not run: no C++ backend is touched (serve/ Python only), so no CUDA/HIP/SYCL
  builds; the live check used the OpenAI route - the Anthropic route and
  `/v1/responses` are covered by the suite's fake-engine tests only

## Merge Danger

**Door:** two-way - revert the one commit; no data, schema, config or feed
migration.

**Blast Radius:** request handling on the three chat routes. A client that sends
none of the three names is unaffected (lookups only, no mutation); the only
behavior change is that a previously-ignored field now does what its name says.

## Limits

- Several aliases at once: the first of the fixed order does (documented in
  DETAILS.md).
- The aliases apply on all three chat routes, so an Anthropic body carrying a
  top-level `thinking_budget` is aliased too, same as the native field's
  behavior there.
- The live check covered one model (IQ3_S) and one prompt family; the routes and
  response shapes are covered by the suite.

Sur le site

Liens install, modèles, releases.