#4932 [Other] serve: preserve_thinking / preserve_reasoning reach the chat template

open · @dan64 · 0 评论 · 去 GitHub 看

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

说明

## Problem and result

A request's `chat_template_kwargs` reaches the template with only `enable_thinking` and `reasoning_effort`.
`preserve_thinking` and `preserve_reasoning` are dropped in `openai_to_messages` / `anthropic_to_messages` before
rendering, so **no client can ask for the reasoning of the earlier turns to be left out of the prompt**. On the Qwen
templates the branch that decides this is `preserve_thinking is undefined or preserve_thinking is true or loop.index0 >
ns.last_query_index`: with the kwarg dropped the first test is always true, so every earlier assistant turn is rendered
with its full `<think>` block, for the whole conversation.

This PR makes both names reach the template, and accepts the same two keys in the `.shared-settings.json` Chat
defaults, for clients that never send the kwarg themselves.

**It does not change a default.** A request that says nothing renders exactly as it does today. What it adds is the
ability to ask for the other behaviour, which did not exist before.

The two names exist because llama.cpp's `--reasoning-preserve` / `--no-reasoning-preserve` set `preserve_reasoning`
(`common/arg.cpp`) while Qwen's templates test `preserve_thinking`; the fixed Qwen templates accept either.

## Related discussion

Same area as #4374 (opt-in prompt shaping in `serve/frontend.py`) and the same class as #843 (an earlier
`serve/frontend.py` rendering fix, accepted): what the server puts into the history decides what the model sees.

I searched the open and closed issues and PRs for `preserve_thinking`, `preserve_reasoning`, `shared-settings` and
`reasoning_effort` before writing this; nothing covers it. If a maintainer would rather have this as an issue first,
say so and I will move it there.

## What it is for, and what it is not

`false` drops the reasoning of the turns **before** the current one and **keeps the current turn's** — the tool-call
loop the model is in the middle of, which is the part that carries the plan across tool results. Measured on the pack's
template (a 2-turn history with a tool call in progress):

```
setting       reasoning of an older turn   reasoning of the current turn
absent        present                      present
true          present                      present
false         absent                       present
```

So this is not "switch the thinking off": the reasoning the model is in the middle of using is untouched whatever the
setting. What `false` removes is the sediment of earlier turns.

## Approach

**`serve/frontend.py`** — `preserve_kwargs()` reads `preserve_reasoning` or `preserve_thinking` from the request's
`chat_template_kwargs` and returns **both** names set to the same value, so a request means the same thing whichever
template the pack carries. Called from both converters, so the OpenAI and Anthropic dialects behave alike. `_flag()`
accepts a JSON boolean or the text form a hand-built body uses (`"false"`, `0`); anything unrecognised reads as off.

**`serve/server.py`** — `preserve_thinking` / `preserve_reasoning` are added to `clean_shared_defaults`' whitelist
(validated as booleans) and `Service.with_shared` fills the value into a request that does not carry one itself; a
request's own value always wins. (Adding the keys to the whitelist also matters for a second reason: see the note at
the end — an unknown key there currently clears **all** the shared settings, which is reported separately as #4933.)

**Compatibility:** opt-in and narrowing only; no wire-format change; tool calls unchanged (checked below).

## Validation

Environment: Windows 11 Pro 26300, i7-12700KF, RTX 5070 Ti 16 GB (sm_120), 111.8 GiB RAM; engine 0.1.51 (CUDA,
compiled from this tree); Qwen3.8-Flash-Next Uncensored Q4_K_S, `--ple-io direct`, `--kv q4_0`, 524288 context.
Server: `serve/server.py --engine strata --config <cfg>.json --port 998x`.
Revision tested: **a2129a95**, on top of upstream `c62d5e3a`. The commit was then amended to **8f8c95ce** to drop
seven unreachable duplicate lines in `preserve_kwargs`; the tests below were **not** re-run after that edit, which removes
dead code and changes no behaviour.

**Ported onto post-0.2.16 main.** 0.2.16 moved `serve/` to `terminal/serve/` and `clean_shared_defaults` into
`terminal/serve/server_parts/http_api.py`, so the branch conflicted and the files this PR edits were no longer at
those paths. Main has been merged in (`3648daf3a`, merge base moved), and the resolved tree is **byte-identical** to
the revision the second block of numbers below was measured on: 5 files, +188/−1, `serve.test_frontend` green on it.
The numbers in the first block, `## Prompt size, fixed transcript`, and everything under *Limits* were measured on
the pre-0.2.16 tree with engine 0.1.51 and the Q4_K_S pack, and are left as they were.

**Passed**

- `python -m pytest serve/test_frontend.py serve/test_server.py -q` — **393 passed, 3 skipped, 661 subtests passed**
  in 116.80 s, including the 11 new tests (`PreserveReasoningSwitch`, `SharedPreserveReasoning`).
- `python -m pytest serve/test_frontend.py serve/test_effort_paths.py serve/test_logprobs.py serve/test_structured.py
  serve/test_tool_routes.py -q` — 93 passed, 175 subtests passed.
- Tool calls on the same revision with the switch active: `finish_reason: tool_calls`, and the call came back as a
  real call — `{"name": "read", "arguments": {"path": "C:\\Windows\\win.ini"}}`.
- Start-up with the shared default in place logs
  `other apps now use the Chat settings: reasoning_effort=medium, preserve_thinking=False`.

**Prompt size, fixed transcript** (the honest number; see *Limits* for why the first figure I published was wrong).
Same conversation text in both arms, identical assistant turns, only the setting differs; `prompt_tokens` from the
response's `usage`:

| turns of history | default (preserve) | `preserve_thinking: false` | saved |
|---|---:|---:|---:|
| 2 | 429 | 91 | 79% |
| 4 | 825 | 149 | 82% |
| 6 | 1221 | 207 | 83% |
| 8 | 1617 | 265 | 84% |

The saving grows with the number of turns, as it should: it is one reasoning block per earlier turn, and that is a
fixed part of each turn's prompt.

**The two checks asked for, on main plus this port** (engine 0.1.63 from the signed 0.2.17 package, SHA-256
`fc986485…dc0f`; IQ3_S; one RTX 5070 Ti; `--kv q4_0`, `SKIP=7`, `PRIOR=0.5`, 524288 context; each live session
from a cold boot). Fixed transcript, the same history in both arms:

| turns of history | default | `preserve_thinking: false` | saved | saved % |
|---:|---:|---:|---:|---:|
| 2 | 1468 | 995 | 473 | 32.2% |
| 4 | 3569 | 2155 | 1414 | 39.6% |
| 6 | 5649 | 3299 | 2350 | 41.6% |
| 8 | 7748 | 4454 | 3294 | 42.5% |

With no kwarg the port renders the same four numbers as main, so the default is untouched; on main the same kwarg
is inert, same transcript and zero tokens of difference. The saving is steady at about **470 tokens per earlier
assistant turn**, so it also grows as a share of the prompt: **32% at two turns, 43% at eight**.

**Conversation cache.** Live 4-turn session, `read` from the serve log: **5.0–5.6k tokens per turn in both arms**.
Dropping the earlier reasoning costs 180–570 extra tokens per turn, 7–16% of what it saves. What stops being
reusable is the tail of the previous generation — the reasoning that just left — not the prefix before it, so the
cache does not re-read from the removed turn on.

**Prefill** is 2163–2182 ms by default and 2196–2225 ms with the drop: 33–43 ms more per turn, on the same ~5k
tokens read in both arms, so part of it is noise. This is not a speed win, and taken at face value it is repaid:
30–45 ms of extra prefill buys 470 tokens less in the prompt on that turn and on every turn after it, compounding
to 3.3k by turn 8 — what pushes the compaction point and the context limit further out.

**Failed:** none observed.

**Not run**

- **Quality.** Whether dropping the earlier reasoning helps or hurts answers on a real workload is not measured here.
  That needs a paired A/B on the same session (`true` vs `false`) and I have not run one. Nothing in this PR should be
  read as a quality claim in either direction.
- **What the conversation cache gives back.** On live 4-turn conversations the arm with `false` reports smaller prompts
  (56/341/429/499 vs 56/480/664/790), but those runs also produced different assistant text, so the difference is not
  attributable to the setting alone. How much of the saving the prefix cache absorbs in a long session is unmeasured.
- **The templates themselves** are not touched. Whether a pack's template honours the kwarg is the template's business;
  on a template that reads neither name the kwargs are passed and have no effect.
- No GPU/CUDA/HIP build is involved: Python only.

## Limits

- **The first figure I put in this PR was misleading and I am replacing it.** It read "2064 -> 52 prompt tokens
  (97.5%)" on a two-turn fixture whose history was almost entirely reasoning and whose last message was a user turn —
  a shape where the current turn has no reasoning to keep, so the whole reasoning left the prompt at once. Real
  conversations end with the assistant's turn in progress, so part of the reasoning stays by design. The table above
  replaces it.
- The reduction is bounded by the template's `last_query_index` rule, not by the server: everything from the last real
  user query onward keeps its reasoning, by design.
- A client that already sends the kwarg keeps its own value, so the shared default cannot force the behaviour on a
  client that asks otherwise. That is deliberate, and it means this is not a policy knob.
- `preserve_reasoning` wins when both are present in the shared settings; the templates use the same precedence.

## A separate defect this run into (reported as #4933, fixed in #4934)

`clean_shared_defaults` raises `ValueError` for a key it does not know, which is right for `POST /settings` (the
client gets a 400 naming the key) but was also used at start-up inside `except (OSError, ValueError): svc.shared = {}`.
A `.shared-settings.json` holding one unrecognised key therefore emptied **every** shared setting, `reasoning_effort`
included, with nothing in the log. Adding these two keys to the whitelist hides that symptom for them; the behaviour
for any other unknown key is #4933, with the fix in #4934. The two PRs touch different lines and do not depend on each
other.

本站相关内容

相关页面的快捷入口。