贡献 / #1966

#1966 serve: continue a final assistant message instead of restating it (#1950)

open · @MistyMoonR · 0 评论 · 去 GitHub 看

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

说明

Fixes #1950.

A request whose last message is an assistant one (the Anthropic API's prefill, and OpenAI clients that do the same) used to get a prompt that closed that turn and opened a fresh one, so the model wrote a new reply that restated the prefill from the top. A stop string then matched the prefill's own text inside that restatement and ended the reply with finish=stop after ~35 characters. Reproduced with the reporter's exact request on Linux as well (the issue is Windows 0.1.42; our box is 2x RTX 4090, v0.1.42 and v0.1.43): the 35-character stub with `stop: ["\ndef"]`, the full body without the stop list.

The template is unchanged and is the same in 0.1.39 (where the reporter says it did not truncate); the prompt shape is the bug, and whether the restatement reaches a stop string varies with the build's numerics. A client that prefills the assistant turn is asking for a continuation, so the prompt should end inside that turn.

**The fix (serve layer only):**
- `Service.continues_final()`: true when the last message is an assistant one with plain, non-empty text and no tool calls; `"effort_position": "end"` configs keep their own prompt shape.
- For such a request `encode_prompt` renders the template without its generation prompt and leaves the turn's closing `<|im_end|>` off, so the prompt ends with the text to continue, inside the assistant's turn.
- The reply's thinking flag is off for these requests (the closed thinking block is already in the prompt), so the parser starts in content state.

**What does not change** (tested): a request whose last message is a user turn, a final assistant message with tool calls, or whitespace-only text gets the exact old prompt; a stop string still cuts the model's own text the moment the model writes it; `StopMatcher` is untouched (a stop string still never runs across a tool call or thinking).

**Verified:**
- 6 new tests (`FinalAssistantContinuation`, byte tokenizer, so the stop strings are split across tokens): the prompt ends with the prefill inside the turn; a stop string in the prefill never ends the reply; a stop string in the continuation still cuts it; the old shapes are unchanged; the Anthropic route continues too. Full serve suite: 712 OK (11 skipped) on 0.1.43 with this patch; the tree without it runs 706.
- End to end on the engine (2x RTX 4090 sm_89, driver 580.95.05, CUDA 13, layer split 25, --pipeline-windows 2, --spec 4, Swift-1.5 Flash-Next GSQ-RCO IQ3_S, KV int8, 262K context, v0.1.43 + this patch): the reporter's request returns the same full body with and without the stop list; a probe that asks for a second function is cut exactly at the model's own `\ndef`; promptfoo suite 6/6; no reasoning recoveries.
- Not tested: 0.1.39 itself (why it usually did not reach the stop string is not investigated); HIP and SYCL builds (the change is Python-only).

本站相关内容

相关页面的快捷入口。