#6481 [Engine] keep the baked qwen think block when preserve_thinking is unset

open · @Mark-1982 · 0 comentários · No GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDALinux

Descrição

## Problem and result

With thinking off, a Qwen3.6 pack writes an empty think block on the first answer, then leaves it out of history. The next prompt is no longer a prefix, so reuse is 0 and a follow-up at about 70k tokens waited 200–213 seconds before the answer started.

After this change the loaded template keeps that block when `preserve_thinking` is unset. The same two short turns went from 0 reused to 36 of 52.

Estimate only, the 70k case was not re-run: the mismatch is a fixed header, so a long follow-up should start in a few seconds instead of about 200. That is roughly 20x to 40x less waiting. The answer's typing speed does not change.

## Related discussion

Fixes #6480
Fixes #5758

#5758 is the same bug and was closed for a missing title prefix. #4932 and #5346 do not rewrite the pack template the server loads.

## Approach

Serve loads `tokenizer/chat_template.jinja` from the pack, not the corrected template in this repo. On load, rewrite only the known broken predicate `preserve_thinking is defined and preserve_thinking is true` to `preserve_thinking is undefined or preserve_thinking is true`. A template that does not contain that line is unchanged.

## Validation

**Passed:**

`python3` unittest of `BakedPreserveThinking` on commit 3142b66 (main), Linux, no GPU. The broken template rendered the empty `<think>` block in history. The repo template was byte-identical after load.

The emitted line, patched into the pack file rather than this Python, was measured on control 0.1.63, 2x RTX 3060 12 GB, pack qwen36-ud-q4_k_xl, thinking off: 48 tokens and 0 reused before, 52 tokens and 36 reused after.

**Failed:**

none observed

**Not run:**

This `frontend.py` was not loaded into the running server. CUDA, HIP and SYCL builds were not run. This change is Python only. The 70k follow-up was not re-run, so the 20x to 40x figure is an estimate from the 200–213 second prefill and the fixed-header mismatch.

## Limits

An assistant turn whose content is empty is still dropped before the template runs. A thinking-only reply is not fixed here.

No site

Links install, modelos, releases.