Issues / #2865
#2865 [Bug]: chat/completions stream ignores stream_options.include_usage (usage stays on the finish_reason chunk)
open · @Eranicus30666 · 0 コメント · GitHub で見る
Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quants
本文
### What happened
OpenAI's streaming contract: with `"stream_options": {"include_usage": true}`, the last chunk before `[DONE]` has `"choices": []` and carries `usage`; the other chunks carry no usage. `/v1/chat/completions` ignores `stream_options` and always puts `usage` on the same chunk as `finish_reason`.
In my setup (LiteLLM in front of Strata), LiteLLM does not read the usage from that chunk and recounts the tokens itself, lower than the engine's count. A client that uses this number to decide when to compact its context then compacts too late.
Expected: with `include_usage: true`, the usage in its own last chunk with `"choices": []`; unchanged without the option. A PR with a fix follows.
Related: #2278 adds the legacy /v1/completions endpoint and says its final chunk already carries the usage; this issue is only about /v1/chat/completions.
### Strata version, GPU, OS
0.1.45 · 2x NVIDIA A2 16GB (sm_86) · Debian 13, CUDA 13.4
関連リンク
インストール・モデル・リリースへの站内リンク。