#4758 [Engine] send the stream role with the first response data

closed · @Tsury · 0 コメント · GitHub で見る

Server & APINVIDIA / CUDASecurityLinux

本文

## Problem and result
`openai_chunks` sends the role-only chunk before the prompt is read. A client that times the first data chunk then measures about 0 ms of prefill (#2391, llama-benchy's TTFR and `est_ppt`). vLLM and llama.cpp send it with the engine's first output.

After this change, the role goes out once, right before the first data chunk. Keep-alive comments are not data and may go first. A progress chunk counts as data.

## Related discussion
#2391. Based on #2480 by @KeyanHu-git, which its author closed as a draft on Oct 22 with no review. The test file comes from that PR. If they would rather carry it on, I will close this one.

## Approach
A small `first()` generator yields the role chunk once, and every path that emits data calls it first.

## Validation
**Passed:**
- New tests against unchanged `main` (870aedd9): 10 failures. With the change: all pass.
- `python -m unittest serve.test_first_chunk serve.test_server serve.test_structured`: 372 tests pass (Linux, Python 3.14).

**Not run:**
- `python -m unittest discover -s serve` and any GPU or model run. This change was split from a maintained v0.2.8 build that runs on an RTX 5090 Laptop; the unit tests above are the evidence for this branch.

## Limits
A client that reads the role chunk as a sign of life gets it later. Keep-alive comments still arrive during a long prompt read.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

関連リンク

インストール・モデル・リリースへのサイト内リンク。