Pull requests / #1840
#1840 serve: a reply that ends inside its thinking with no answer is continued (#1814)
open · @47Hunter47 · 0 Kommentare · Auf GitHub
Server & APINVIDIA / CUDAModels & quants
Beschreibung
Fixes #1814. I reproduced it on an RTX 3090 with Qwen3.8-Flash-Next IQ3_XXS on 0.1.41 and dumped the raw turns with `STRATA_DEBUG=1`. ## What I see The model is about to write its closing thinking tag and the **turn-end token comes instead**: ``` [strata] raw: 'We need to respond to user: "In a markdown code fence, show exactly how the closing thinking tag looks ... inside prose."\n\nNeed likely show closing thinking tag: <|im_end|>' ``` `finish_reason` is `stop`, the reasoning stops mid-sentence, no answer is produced, and the client gets an empty reply. My trigger runs (temperature 0, production config: 250K context, `--kv int8`, MTP speculative decoding): | runs | empty replies | | --- | --- | | first set, 9 runs (streaming and whole, `STRATA_PREFILL_CPU_SHARE` on and off) | 6 of 9 | | same prompts, later run, 5 runs | 3 of 5 | | same prompts again, later run, 6 runs | 0 of 6 | So it is not deterministic: the same prompt can answer normally on one pass and end with nothing on the next. The prompt has to make the model reach for its closing tag (asking it to show, parse or quote the tag), and the cut lands right where the tag would start, or right after a backtick - which matches what danirebollo describes in the issue. Five real turns are in `serve/fixtures/literal_think_specimens.json`. Each carries a `kind`: `no-answer` is this bug, `answered` reached its answer, `length-cut` ended on max_tokens instead. ## Why the parser cannot catch it In this model's tokenizer the tag is not a special token: it is eight ordinary text tokens (`<`, `/`, `t`, `h`, `i`, `n`, `k`, `>`), and the stop token that ends the turn is the ordinary end-of-turn token. A turn that ends inside its thinking therefore looks exactly like a turn that ended inside its thinking - there is nothing in the stream that says "I meant that". That is also why #537's mark/unmark work (the input side) does not cover it. ## The fix: two layers, both behind one opt-in flag **1. The end is reported** (with the flag on, like the continuation): * one log line per reply: `the reply ended inside its thinking with no answer (N tokens generated)` - with the guard on the continuation line follows it, and once its cap is reached the line says so * `totals.ended_inside_thinking`, exposed as `/metrics`' `totals_ended_inside_thinking_total` * `"ended_inside_thinking": true` beside `choices` in the response, and in the stream's last chunk (the same place `timings` goes) So a client that turned the guard on and got an empty reply is told why it is empty and can retry, and an operator sees the rate. The field is absent on a normal reply. **2. The continuation** (`"literal_think_guard": true` in `strata-<model>.json`, or `STRATA_LITERAL_THINK_GUARD=1`), like #1053: the reply is continued with the thinking closed the way #123's wrap-up closes it (`REASONING_CLOSE`), so the next pass's prompt is the one that produced the reply plus what it generated plus that close, and the engine continues from the prefix it already holds. `parser.state == "content"` is the same end from the other side - the model wrote the tag as text, the parser took it as the marker and nothing followed - and there the tag goes back into the reasoning text first (`OutputParser.reopen_reasoning()`). It is capped per reply (`LITERAL_THINK_RETRIES = 2`) with a `literal_think_retries` counter in the request trace, so a model that keeps doing it cannot loop. Both layers are behind the same switch, like #1053: with the flag off a server behaves exactly as it did before #1814 - no log line, no total, no `/metrics` series, no field on the response - so reporting and continuation are one deliberate choice, and the reporting can stay a proof of concept until it is approved on its own. They work together: whichever fires first sets `close_retried`. If you would rather have either on by default, the default is one line (`Service.literal_think_guard`). ## Tests `serve/test_literal_think_guard.py` replays the five real turns through a mock engine: every `no-answer` specimen gets the answer with one continuation, the flag-off case carries nothing (no log, no total, no field) and stays empty, the answer the guard reached is reported too, a reply that hit the cap is reported and still empty, an `answered`/`length-cut` specimen is not touched, a normal reply carries no flag, the flag is off by default, and the cap holds. `serve/test_prometheus.py` checks the `/metrics` series does not exist until the guard counts. `python -m unittest discover -s serve -t . -p "test_*.py"` -> 707 tests, 11 skipped. The only failure I see is `test_keepawake`'s `test_service_holds_for_the_whole_request_including_early_close`: it re-runs alone as 3 FAILED / 2 OK (its `wait_for(..., t=3.0)` races the keepawake thread's release), so it is not from this change. The `test_responses.py` telemetry `(_loop)` thread count I mentioned earlier is gone here - this branch's own test file now waits for the sampler it starts. I have it running on this machine (one 3090, IQ3_XXS, the same config the issue reports) with the flag on.
Mehr auf der Site
Links zu Install, Modellen, Releases.