#6481 [Engine] keep the baked qwen think block when preserve_thinking is unset
open · @Mark-1982 · 0 commentaires · Sur GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDALinux
Description
## Problem and result With thinking off, a Qwen3.6 pack writes an empty think block on the first answer, then leaves it out of history. The next prompt is no longer a prefix, so reuse is 0 and a follow-up at about 70k tokens waited 200–213 seconds before the answer started. After this change the loaded template keeps that block when `preserve_thinking` is unset. The same two short turns went from 0 reused to 36 of 52. Estimate only, the 70k case was not re-run: the mismatch is a fixed header, so a long follow-up should start in a few seconds instead of about 200. That is roughly 20x to 40x less waiting. The answer's typing speed does not change. ## Related discussion Fixes #6480 Fixes #5758 #5758 is the same bug and was closed for a missing title prefix. #4932 and #5346 do not rewrite the pack template the server loads. ## Approach Serve loads `tokenizer/chat_template.jinja` from the pack, not the corrected template in this repo. On load, rewrite only the known broken predicate `preserve_thinking is defined and preserve_thinking is true` to `preserve_thinking is undefined or preserve_thinking is true`. A template that does not contain that line is unchanged. ## Validation **Passed:** `python3` unittest of `BakedPreserveThinking` on commit 3142b66 (main), Linux, no GPU. The broken template rendered the empty `<think>` block in history. The repo template was byte-identical after load. The emitted line, patched into the pack file rather than this Python, was measured on control 0.1.63, 2x RTX 3060 12 GB, pack qwen36-ud-q4_k_xl, thinking off: 48 tokens and 0 reused before, 52 tokens and 36 reused after. **Failed:** none observed **Not run:** This `frontend.py` was not loaded into the running server. CUDA, HIP and SYCL builds were not run. This change is Python only. The 70k follow-up was not re-run, so the 20x to 40x figure is an estimate from the 200–213 second prefill and the fixed-header mismatch. ## Limits An assistant turn whose content is empty is still dropped before the template runs. A thinking-only reply is not fixed here.
Sur le site
Liens install, modèles, releases.