Issues / #2906

#2906 [Bug]: reasoning_loop_recovery: recover never fires for a low or medium request (focused_recovery_prompt only matches the template's xhigh sentence)

open · @BoogeAI · 0 commentaires · Sur GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

### What happened

With `"reasoning_loop_recovery": "recover"` in a model config, a reply whose thinking repeats whole passages **is detected and then does nothing**: the server logs the detection and the thinking carries on to `reasoning_budget_tokens` (or to `max_tokens` when no budget is set). The reply is not continued with the low-effort instruction, and nothing says why.

Measured on 0.1.44.1 and again on 0.1.45, a community qwen4exp fine-tune (RVN abliterated IQ3_S), `reasoning_effort: low`, one prompt that makes the model re-verify the same facts, `max_tokens` 8192, `reasoning_budget_tokens` 8192 then 4096:

| run | budget | detection | what happened next |
| --- | --- | --- | --- |
| 1 | 8192 | `repeated reasoning detected at 3072 tokens (coverage=0.277)` | thought on to 6144, then the budget wrapped it up |
| 2 | 8192 | `repeated reasoning detected at 3072 tokens (coverage=0.289)` | the same |
| 3 | 4096 | `repeated reasoning detected at 3072 tokens (coverage=0.343)` | thought on to 4096, then the budget wrapped it up |

So the detector works (it is the log line above); the recovery step is what never runs.

### Why

`focused_recovery_prompt()` (`serve/server.py:196` in 0.1.45) builds the spliced prompt by replacing **the template's own xhigh sentence** (`HIGH_EFFORT`, line 160) in the first system message with the low one, and returns `None` when that sentence is not there as whole tokens:

```python
old, new = tok.encode(HIGH_EFFORT, parse_special=False), tok.encode(LOW_EFFORT, parse_special=False)
for i in range(len(head) - len(old) + 1):
    if head[i:i + len(old)] == old:
        return ids[:i] + new + ids[i + len(old):] + list(generated)
return None
```

With `"effort_position": "start"` (the default) a request that asks for **low** or **medium** is rendered with *that level's* sentence at the top of the first system message, so the xhigh sentence is absent and the function returns `None`. The caller only acts on a non-None result:

```python
recover_prompt = focused_recovery_prompt(self.tok, ids, raw_ids)
if recover_prompt is not None:
    break
```

`recover` is therefore a silent no-op for every low/medium request — which is the common case as soon as a config sets `"reasoning_effort": "low"` (as mine does, to keep a local model from burning its output budget on thinking) or a client picks Low in its own selector. `"reasoning_loop_recovery": "stop"` is unaffected (it does not use this function and did end the reply in my runs).

### How I checked it (no engine needed)

Rendering one request both ways with the server's own functions and then calling `focused_recovery_prompt` on it:

```
--- effort_position="start" (default)
    prompt tokens                : 63
    xhigh sentence in the prompt : False
    low   sentence in the prompt : True
    focused_recovery_prompt()    : None -> the recover branch cannot fire
--- effort_position="end"
    prompt tokens                : 105
    xhigh sentence in the prompt : True
    low   sentence in the prompt : True
    focused_recovery_prompt()    : 93 ids -> it can fire
```

Script: build the tokenizer and the `ChatTemplate` from a pack's `tokenizer/` exactly as `main()` does, then

```python
low_text  = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=True)          # start
hist      = tpl.render(msgs, add_generation_prompt=False)                                 # end, default (xhigh)
full      = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=True)
mine      = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=False)
end_text  = hist + EFFORT_TURN.format(EFFORT_TEXT["low"]) + full[len(mine):]
for t in (low_text, end_text):
    print(focused_recovery_prompt(tok, tok.encode(t, parse_special=True), []))
```

### Workaround I use now

`"effort_position": "end"` in the model config: the history is then rendered with the template's default (xhigh) and the real level follows in a short system turn before the answer, so the xhigh sentence is in the prompt and the recovery can splice it. With it, the detection line is followed by `resuming the same output with the low-effort instruction (reasoning_loop_recovery)` (measured twice, coverage 0.277 and 0.343).

Two honest notes on that: the nudge is not a cure — in both runs the model went on thinking for another ~1000-3000 tokens to the budget — and `effort_position: "end"` is a separate opt-in with its own behaviour (the engine checkpoints in front of the tail turn).

### Suggested fix

Pass the request's level into `focused_recovery_prompt()` and look for *that* sentence (the template knows `low`, `medium`, `xhigh`; `high` maps to xhigh), or fall back to the low sentence when the level's own is not found. If neither is found, print one line saying the recovery was skipped and why, instead of failing silently — the current behaviour costs a full thinking budget with no hint.

### Environment

0.1.44.1 and 0.1.45, prebuilt CUDA 13.0 engine (sm_75/86/89/120) · RTX 5080 16 GB · Windows 11 26200 · Core Ultra 7 270K Plus, 95 GB RAM. Config: `reasoning_effort: low`, `reasoning_budget_tokens: 4096/8192`, `reasoning_loop_recovery: recover`, `reasoning_close_retry: true`.

### Not tested

AMD HIP and SYCL (CUDA only here); a high/xhigh request under `effort_position: "start"` (that one renders the xhigh sentence at the top, so the recovery should fire — I did not run it); the Anthropic route's `output_config.effort` path; and whether the model's repetition after the nudge would end on its own with a different prompt.

Sur le site

Liens install, modèles, releases.