Pull requests / #2480
#2480 serve: send the role chunk with the first response data
open · @KeyanHu-git · 0 コメント · GitHub で見る
本文
## Title serve: send the role chunk with the first response data Issue: Resolves #2391 ## Summary The streaming chat endpoint sends its role-only chunk before starting prompt processing. Clients that measure latency from the first SSE data chunk therefore record a near-zero prompt time. This is the source of the inflated prefill rate reported by llama-benchy in #2391. Send the role chunk immediately before the first response data instead. Keep-alive comments still arrive while the engine reads the prompt, and clients requesting `return_progress` still receive progress updates. ## What changed `openai_chunks` opens the stream when it emits content, reasoning, a tool call, MCP data or requested prompt progress. The role remains the first chunk and appears once. Empty content/reasoning events emit nothing; an empty answer sends the role before its finish chunk. An engine error before the first output goes straight to the existing error handler. ## Extra Notes Twelve focused tests cover text and reasoning, empty events and answers, streamed and complete tool calls, requested progress, MCP rounds, errors and non-streamed collection. Ten fail against the original server; all twelve pass with the change. The progress and MCP cases verify that requested data can open the stream before an answer without sending another role chunk later. On macOS and Windows, a separate HTTP/SSE check holds a mock engine in prefill and reads the live response. The original server sends data before prefill is released. The patched server sends keep-alive comments until release, then the role and answer in order. This checks the response timing contract without treating mock-engine timings as model performance. The full service suite passes on macOS: 786 tests and 5638 subtests, with 11 skips. The twelve focused tests also pass on Windows.
関連リンク
インストール・モデル・リリースへの站内リンク。