问题 / #458

#458 Changing reasoning_effort re-reads the whole prompt: the effort text is the first thing the template renders

closed · @sebastianrcnt · 3 评论 · 去 GitHub 看

BenchmarksServer & APINVIDIA / CUDAModels & quants

说明

### What happens

`serve/chat_template.jinja` puts the reasoning effort instructions at the very top of the first system turn (`<|im_start|>system\nReasoning effort is set to xhigh. ...`). The conversation cache reuses the longest common token prefix, so a request that only changes `reasoning_effort` (or turns thinking off) differs from the previous one a few tokens in and re-reads the entire prompt.

Agent clients do this routinely. OpenCode's variants, a "think harder" toggle, a subagent with a different effort, `none` for a quick tool step. On a 150K-token agent session that is a full prefill (~100 s at 1.5K tok/s on a 3090) just for switching the level.

### Measured (no GPU needed)

The same 22K-token conversation (system prompt + 20 turns) rendered with this repo's template at `xhigh` vs. the other levels, tokenized with the Strata tokenizer; "first differing token" is the most the cache can reuse:

| template | prompt tokens | low vs xhigh | medium vs xhigh | thinking off vs xhigh |
|---|---|---|---|---|
| current (`origin/main`) | 22,144 | **9** | **3** | **3** |
| effort in a trailing system turn (below) | 22,148 | 22,110 | 22,102 | 22,102 |

### What I'm running locally (works, but it needs both halves)

1. Template: drop `reasoning_instructions` from the top system turn and emit it as a short system turn right before the generation prompt:
   ```jinja
   {%- if add_generation_prompt %}
       {%- if reasoning_instructions %}
           {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
       {%- endif %}
       {{- '<|im_start|>assistant\n' }}
   ```
2. Engine: the turn checkpoint goes at the *last* `<|im_start|>`, which is now the assistant header *after* that trailing block - so with the template change alone the checkpoint holds the effort text and even a normal next turn reuses nothing (I measured 0 reused on a plain follow-up with stock 0.1.34 + the new template). A small rule moves it in front of the trailing system turn (`--tail-role-token <id of "system">`, 8678 for this tokenizer):
   ```cpp
   if (turn_at > 0 && o.tail_role_token >= 0)
       for (int64_t i = turn_at - 1; i > resume; --i)
           if (ids[(size_t) i] == o.turn_token) {
               if (ids[(size_t) i + 1] == o.tail_role_token) turn_at = i;   // <|im_start|>system right before
               break;
           }
   ```
   (It could just as well be the default whenever the turn before the last header is a system turn, without a flag.)

With both, on an RTX 3090 (OrcaRouter IQ3_XXS, 262K, int8 KV): switching effort on an 8,222-token prompt reused 8,155 tokens; seven requests in a row on a ~5K prompt, each with a different effort value (`extreme`, `minimal`, `none`, `low`, `high`, and two non-standard ones that the template maps to its default), reused 5,005-5,047 tokens on every request after the first, and normal follow-up turns keep reusing as before.

### The open question

This changes what the model sees: the effort text moves from the top of the system prompt to just before the answer. I have not measured whether that changes how well the level is followed or answer quality (the official Qwen template puts it at the top). Would you take a PR along these lines (perhaps opt-in first) or do you prefer another approach (e.g. a checkpoint right after the effort line, so only the system prompt is re-read)?

本站相关内容

相关页面的快捷入口。