Issues / #2141

#2141 serve: thinking_budget_tokens (llama-server's name) is ignored, so llama.cpp clients get unbounded thinking

open · @xrip · 0 comments · View on GitHub

Server & APIMulti-GPUModels & quants

Description

## Summary

Strata's server ignores `thinking_budget_tokens`, the name llama-server also accepts for a request's thinking budget. A
client written for llama-server that sends it gets **no thinking budget at all**, and nothing in the log or the response
says so.

## Expected

The same as llama-server, which reads `reasoning_budget_tokens` and falls back to `thinking_budget_tokens`
(`tools/server/server-common.cpp`):

```cpp
int reasoning_budget = json_value(body, "reasoning_budget_tokens",
                       json_value(body, "thinking_budget_tokens", -1));
```

## Actual

`Service.reasoning_budget()` (`serve/server.py`) reads only `reasoning_budget_tokens`. With `thinking_budget_tokens` alone
the thinking is unbounded: it stops only at `max_tokens` or at the end of the context.

## Impact

Any client that targets llama-server's API and relies on `thinking_budget_tokens`. For example, the pi coding agent's
llama.cpp provider ([pi-llama-cpp](https://github.com/gsanhueza/pi-llama-cpp)) sends it for every request: 8,192 tokens at
pi's default `medium` level.

What it cost on our box (2x CMP 50HX, Qwen3.8-Flash-Next IQ3_XXS):
- **The request:** pi's automatic compaction, a summarization over a 109,013-token serialized history. pi caps that
  answer at 0.8 x `reserveTokens`, 26,214 tokens here.
- **What happened:** the model thought until that cap and never wrote the summary. pi retried the identical request five
  times, about 250 s each (21 minutes), and the session was not compacted.
- **Engine log, each attempt:** `prompt 109013 tokens = 109008 reused + 5 read ..., 26214 generated in ~250000 ms`.

Even where nothing fails outright, every request from such a client thinks without the limit the user chose.

## Reproduce

```sh
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "m", "max_tokens": 4000, "thinking_budget_tokens": 64,
  "messages": [{"role": "user", "content": "Think step by step: how many primes are below 200?"}]}'
```

The `reasoning_content` runs far past 64 tokens. With `"reasoning_budget_tokens": 64` it is wrapped up at 64.

## Fix

#2140: `reasoning_budget()` also reads `thinking_budget_tokens`, and `reasoning_budget_tokens` wins when both are set.
It comes with a test.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.