Issues / #458
#458 Changing reasoning_effort re-reads the whole prompt: the effort text is the first thing the template renders
closed · @sebastianrcnt · 3 comentários · No GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quants
Descrição
### What happens
`serve/chat_template.jinja` puts the reasoning effort instructions at the very top of the first system turn (`<|im_start|>system\nReasoning effort is set to xhigh. ...`). The conversation cache reuses the longest common token prefix, so a request that only changes `reasoning_effort` (or turns thinking off) differs from the previous one a few tokens in and re-reads the entire prompt.
Agent clients do this routinely. OpenCode's variants, a "think harder" toggle, a subagent with a different effort, `none` for a quick tool step. On a 150K-token agent session that is a full prefill (~100 s at 1.5K tok/s on a 3090) just for switching the level.
### Measured (no GPU needed)
The same 22K-token conversation (system prompt + 20 turns) rendered with this repo's template at `xhigh` vs. the other levels, tokenized with the Strata tokenizer; "first differing token" is the most the cache can reuse:
| template | prompt tokens | low vs xhigh | medium vs xhigh | thinking off vs xhigh |
|---|---|---|---|---|
| current (`origin/main`) | 22,144 | **9** | **3** | **3** |
| effort in a trailing system turn (below) | 22,148 | 22,110 | 22,102 | 22,102 |
### What I'm running locally (works, but it needs both halves)
1. Template: drop `reasoning_instructions` from the top system turn and emit it as a short system turn right before the generation prompt:
```jinja
{%- if add_generation_prompt %}
{%- if reasoning_instructions %}
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
{%- endif %}
{{- '<|im_start|>assistant\n' }}
```
2. Engine: the turn checkpoint goes at the *last* `<|im_start|>`, which is now the assistant header *after* that trailing block - so with the template change alone the checkpoint holds the effort text and even a normal next turn reuses nothing (I measured 0 reused on a plain follow-up with stock 0.1.34 + the new template). A small rule moves it in front of the trailing system turn (`--tail-role-token <id of "system">`, 8678 for this tokenizer):
```cpp
if (turn_at > 0 && o.tail_role_token >= 0)
for (int64_t i = turn_at - 1; i > resume; --i)
if (ids[(size_t) i] == o.turn_token) {
if (ids[(size_t) i + 1] == o.tail_role_token) turn_at = i; // <|im_start|>system right before
break;
}
```
(It could just as well be the default whenever the turn before the last header is a system turn, without a flag.)
With both, on an RTX 3090 (OrcaRouter IQ3_XXS, 262K, int8 KV): switching effort on an 8,222-token prompt reused 8,155 tokens; seven requests in a row on a ~5K prompt, each with a different effort value (`extreme`, `minimal`, `none`, `low`, `high`, and two non-standard ones that the template maps to its default), reused 5,005-5,047 tokens on every request after the first, and normal follow-up turns keep reusing as before.
### The open question
This changes what the model sees: the effort text moves from the top of the system prompt to just before the answer. I have not measured whether that changes how well the level is followed or answer quality (the official Qwen template puts it at the top). Would you take a PR along these lines (perhaps opt-in first) or do you prefer another approach (e.g. a checkpoint right after the effort line, so only the system prompt is re-read)?
No site
Links install, modelos, releases.