#5346 [Other] serve: preserve multi-turn reasoning by default and skip empty historical think blocks
open · @bsorensen110 · 0 コメント · GitHub で見る
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux
本文
## Problem and result
Two things in the multi-turn reasoning path, found while investigating reasoning degradation on long agentic conversations with Qwen3.8-Flash-Next.
**1. A prompt-serialization bug: historical assistant turns with empty reasoning are rendered as an empty think block.** `serve/chat_template.jinja` decides to keep a historical turn's thinking on `preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index` — it never checks that `reasoning_content` is non-empty. A client that echoes an assistant turn back with `reasoning_content: ""` — which is what happens whenever the reasoning item is present but its content was dropped (e.g. `/v1/responses` history where the reasoning summary is `[]`, or clients that strip reasoning text to save tokens) — gets this into the next prompt:
```
<|im_start|>assistant
<think>
</think>
first answer<|im_end|>
```
So the conversation history accumulates one empty `<think>\n\n</think>` block per reasoning-less assistant turn — a few-shot example of "reason by saying nothing", right in the few-shot region of every subsequent request. After the fix the same history renders `<|im_start|>assistant\nfirst answer` — the empty block is simply not emitted. Turns *with* reasoning are rendered exactly as before, and the current turn's think-open is untouched.
**2. `preserve_reasoning` / `preserve_thinking` had no plumbing through the serve front end.** llama.cpp's name is `preserve_reasoning`, Transformers/Qwen templates test `preserve_thinking`, and there was no way to reach either: a request's `chat_template_kwargs` entries were dropped before rendering, `/v1/responses` had no path at all, the shared Chat settings rejected the keys, and `ChatTemplate._detect_caps()` inferred template support from one default render. After the fix:
- both names (and the inverse `clear_thinking`) are accepted on `/v1/chat/completions`, `/v1/anthropic` messages and `/v1/responses`, at request level or inside `chat_template_kwargs`, and reach the template normalized to both spellings;
- the shared `.shared-settings.json` accepts `preserve_reasoning` / `preserve_thinking` / `clear_thinking` (booleans) as defaults, and a request's own value always wins;
- capability detection is now differential: the probe renders the same history with the switch on and off, and the template is credited with `supports_preserve_reasoning` only when the reasoning block actually appears on and disappears off;
- preservation defaults to on **only** for templates that pass that probe — a template that does not advertise support renders exactly as it does today, with no new kwarg noise;
- in `openai_to_messages`, `chat_template_kwargs.enable_thinking: false` used to reset the whole kwargs dict, discarding whatever else had been computed; it now merges, so the two settings coexist.
## Related discussion
- **#4932** (dan64) covers overlapping ground on point 2: it plumbs `preserve_thinking` / `preserve_reasoning` from `chat_template_kwargs` and the shared settings through the OpenAI/Anthropic converters, without changing a default. This PR does the same job for the same clients **plus** the `/v1/responses` path, the `clear_thinking` alias, the differential caps probe and the template guard in point 1, but it also turns the default on for templates that prove support — so if the maintainer prefers the strict opt-in of #4932, the two can be merged by keeping that PR's defaults policy and adding this one's template guard + responses path. Expect conflicts in `serve/frontend.py` / `serve/server.py`.
- **#5218** reports the same symptom class (degenerate long-run reasoning on agent prompts) from the detector side; this PR is independent of it and neither prevents nor duplicates its guard.
## Approach
- `serve/chat_template.jinja` — one condition: `{%- if reasoning_content and (preserve_thinking is undefined or ...) %}`. Empty reasoning never opens a think block.
- `serve/frontend.py` — `_normalize_preserve_reasoning()` (accepts both spellings and their text forms; `preserve_thinking` wins when both are given at one level), `preserve_reasoning_kwargs()` (extracts from a request and/or its `chat_template_kwargs`, maps the inverse `clear_thinking`), called from `ChatTemplate.render()` and both request converters.
- `serve/responses.py` — `template_kwargs()` merges the shared defaults first and request-level values on top.
- `serve/server.py` — the three keys join `SHARED_KEYS` (validated as booleans), `Service.with_shared()` fills them into a request that carries none, and `_responses_prepare` now passes `svc.shared` to `template_kwargs()`.
- `serve/test_preserve_reasoning.py` — the serialization regression test, plus a `PreserveReasoningKwargs` case covering the aliases, text forms, `clear_thinking`, both request converters and the `/v1/responses` shared/request merge; `serve/test_server.py` gains a case for the shared default, request override and boolean validation, and its `LiteralThinkTags` vector drops one open/close token-id pair — the old count was counting the empty-think-block artifact this PR removes. `serve/chat_golden.json` vectors updated where the guard changes rendering (one historical tool-call turn no longer serializes `<think>\n\n</think>` before its call).
## Validation
Revision: this branch, one commit (`3e94381a9`) on upstream main `9f7d977d2` (0.2.10 line). Pure Python path — `jinja2` template rendering with `MockEngine`/`ByteTokenizer`, no GPU or model files involved. Environment: Linux, Python 3.14, pytest.
**Passed**
- **RED→GREEN for the serialization bug.** With `serve/test_preserve_reasoning.py` against the **base** template: `AssertionError: '<think>\n\n</think>' unexpectedly found in '<|im_start|>system ... <|im_start|>assistant\n<think>\n\n</think>\n\nfirst answer<|im_end|> ...'`. With the guard: `1 passed`.
- **RED for the plumbing.** With the new tests against base `serve/frontend.py`: `ImportError: cannot import name '_normalize_preserve_reasoning'` — the normalization the tests exercise does not exist on main.
- Focused suites: `python -m pytest serve/test_preserve_reasoning.py serve/test_frontend.py serve/test_responses.py serve/test_effort_paths.py -q` → **81 passed, 1 skipped, 110 subtests passed**.
- Full regression: `python -m pytest serve/ tools/test_strata_tokenizer.py -q` → **1449 passed, 25 skipped, 14178 subtests passed**, with **1 failed**: `serve/test_keepawake.py::KeepAwakeTests::test_service_holds_for_the_whole_request_including_early_close`. That failure is pre-existing: it reproduces identically (same assertion, `AssertionError: False is not true` on the power-keepalive call list) in a detached worktree of clean upstream `9f7d977d2` on this machine, and none of the files this PR touches are on the keep-awake path.
**Failed:** none attributable to this change — the single failure above is the pre-existing baseline case, identical on clean main.
**Not run**
- **Answer quality.** Whether preserving earlier turns' reasoning measurably helps or hurts multi-turn agent answers is *not established* — a small repeated-fixture study run against a live server was inconclusive. Nothing here should be read as a quality claim in either direction; this fixes what reaches the prompt, consistently and on all three API dialects, not a measured outcome.
- Device runs on CUDA/HIP engines: the changed files are Python-level prompt construction; no engine path is touched.
## What this does NOT fix
- The **model-pack** copies of the Qwen chat template (the `chat_template.jinja` shipped inside tokenizer packs) carry the same unguarded line. This PR fixes the serve fallback template; packs need the same one-line guard at pack build time. (Verified on an installed Qwen3.8-Flash-Next IQ3_S pack template: same empty-think render.)
- Degenerate or pathologically long historical reasoning blocks are fed back as-is; no sanitization.
- Opaque encrypted reasoning produced by other providers (not prefixed `strata.r1:`) still cannot be replayed on `/v1/responses` — by design.
- The default-on applies only where the differential probe proves the template honors the switch; on everything else behavior is byte-identical to today.
## Test plan
- [x] New serialization regression test: RED on the base template, GREEN with the guard
- [x] Plumbing tests: ImportError-RED on base frontend, GREEN with the fix
- [x] Focused suites + full `serve/` + tokenizer golden vectors pass (counts above)
- [ ] CI
## Reproduction
```sh
# from the repo root, on main: the empty think block appears
python - <<'EOF'
from serve.frontend import ChatTemplate
t = ChatTemplate("serve/chat_template.jinja")
history = [
{"role": "user", "content": "first"},
{"role": "assistant", "content": "first answer", "reasoning_content": ""},
{"role": "user", "content": "second"},
]
p = t.render(history)
print("EMPTY THINK BLOCK PRESENT:", "<think>\n\n</think>" in p) # main: True, this PR: False
EOF
```
関連リンク
インストール・モデル・リリースへのサイト内リンク。