改动 / #2870

#2870 serve: put the streamed usage in its own final chunk when include_usage is requested

open · @Eranicus30666 · 0 评论 · 去 GitHub 看

Setup & installServer & APIMulti-GPUAMD / HIPSecurity

说明

## Title
Issue: Resolves #2865

## Summary
With `stream_options.include_usage`, the streamed usage now goes in its own final chunk with `"choices": []`, as in OpenAI's streaming contract. Without the option nothing changes.

## What changed
- `serve/server.py` (+10): in the streaming loop of `/v1/chat/completions`, a `split_usage` flag from `stream_options.include_usage`. When it is set and a chunk carries `usage`, that chunk is written as two chunks: the same chunk without `usage`, then a chunk with `"choices": []` and the usage.
- `serve/test_server.py` (+59): three tests (usage alone in the last `choices: []` chunk with the option; usage unchanged on the `finish_reason` chunk without it; same numbers as the non-streaming reply).

Left alone: non-streaming replies; requests without the option (the new branch is skipped and the original write is used); `/v1/messages`; `/v1/responses`; the legacy completions endpoint of #2278.

## Extra Notes
How I checked:
- The new tests and the whole `serve/test_server.py` suite pass (340 tests) on v0.1.45 + this commit.
- Live, on engine 0.1.45 (2x NVIDIA A2, UD-Q4_K_XL): a streamed request with the option puts the usage alone in the last `choices: []` chunk; without the option it stays on the `finish_reason` chunk.
- The same change has run in my production setup since 11/10 (on 0.1.39).

Not checked: HIP/SYCL (Python-only change, no engine file touched), clients other than LiteLLM, and I compared the chunk structure rather than a byte diff for the default path.

Co-authored with Claude.

本站相关内容

相关页面的快捷入口。