问题 / #2865

#2865 [Bug]: chat/completions stream ignores stream_options.include_usage (usage stays on the finish_reason chunk)

open · @Eranicus30666 · 0 评论 · 去 GitHub 看

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quants

说明

### What happened

OpenAI's streaming contract: with `"stream_options": {"include_usage": true}`, the last chunk before `[DONE]` has `"choices": []` and carries `usage`; the other chunks carry no usage. `/v1/chat/completions` ignores `stream_options` and always puts `usage` on the same chunk as `finish_reason`.

In my setup (LiteLLM in front of Strata), LiteLLM does not read the usage from that chunk and recounts the tokens itself, lower than the engine's count. A client that uses this number to decide when to compact its context then compacts too late.

Expected: with `include_usage: true`, the usage in its own last chunk with `"choices": []`; unchanged without the option. A PR with a fix follows.

Related: #2278 adds the legacy /v1/completions endpoint and says its final chunk already carries the usage; this issue is only about /v1/chat/completions.

### Strata version, GPU, OS

0.1.45 · 2x NVIDIA A2 16GB (sm_86) · Debian 13, CUDA 13.4

本站相关内容

相关页面的快捷入口。