Pull requests / #2870
#2870 serve: put the streamed usage in its own final chunk when include_usage is requested
open · @Eranicus30666 · 0 Kommentare · Auf GitHub
Setup & installServer & APIMulti-GPUAMD / HIPSecurity
Beschreibung
## Title Issue: Resolves #2865 ## Summary With `stream_options.include_usage`, the streamed usage now goes in its own final chunk with `"choices": []`, as in OpenAI's streaming contract. Without the option nothing changes. ## What changed - `serve/server.py` (+10): in the streaming loop of `/v1/chat/completions`, a `split_usage` flag from `stream_options.include_usage`. When it is set and a chunk carries `usage`, that chunk is written as two chunks: the same chunk without `usage`, then a chunk with `"choices": []` and the usage. - `serve/test_server.py` (+59): three tests (usage alone in the last `choices: []` chunk with the option; usage unchanged on the `finish_reason` chunk without it; same numbers as the non-streaming reply). Left alone: non-streaming replies; requests without the option (the new branch is skipped and the original write is used); `/v1/messages`; `/v1/responses`; the legacy completions endpoint of #2278. ## Extra Notes How I checked: - The new tests and the whole `serve/test_server.py` suite pass (340 tests) on v0.1.45 + this commit. - Live, on engine 0.1.45 (2x NVIDIA A2, UD-Q4_K_XL): a streamed request with the option puts the usage alone in the last `choices: []` chunk; without the option it stays on the `finish_reason` chunk. - The same change has run in my production setup since 11/10 (on 0.1.39). Not checked: HIP/SYCL (Python-only change, no engine file touched), clients other than LiteLLM, and I compared the chunk structure rather than a byte diff for the default path. Co-authored with Claude.
Mehr auf der Site
Links zu Install, Modellen, Releases.