#6105 OpenAI streaming: opt-in continuous_usage_stats (vLLM-style cumulative usage on chunks)
open · @woniubility · 0 commentaires · Sur GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDASecurity
Description
## What changes - Adds opt-in `stream_options.continuous_usage_stats` to `/v1/chat/completions`: when `stream` + `include_usage` + `continuous_usage_stats` are all true, every data chunk carries the request's **cumulative** usage (`prompt_tokens` / `completion_tokens` / `total_tokens`), so clients that compute a live rate from streamed usage (e.g. a coding agent's thinking-speed badge) update while the model is still generating. - The count is `Service._run`'s own `n` — thinking, tool-call argument tokens, wrap-up/close-retry tokens and every MCP round are included and never reset mid-request. It is not a text-length, SSE-chunk-count or re-tokenization estimate, so mid-stream values can never disagree with the final usage. - A per-request `progress` dict is created only for requests that ask for the option, shared by the MCP run and `openai_chunks()`; the final chunk's authoritative usage (with accurate `prompt_tokens_details` and `timings`) overrides the mid-stream snapshot, which omits cache details it cannot know yet. ## What remains unchanged - Default streams and final usage accounting (a request without `continuous_usage_stats` - or with only `include_usage` - is byte-identical to before, usage only on the final chunk, as today even without `include_usage`). - Model output, sampling, detokenization, tool parsing and engine decode. - Anthropic `/v1/messages` and Responses `/v1/responses`. ## Motivation Clients like omp show a real-time tok/s badge for streaming thinking, computed as `usage` deltas between chunks. Today Strata only reports usage on the final chunk, so the badge stays hidden until generation ends; vLLM's `continuous_usage_stats` already feeds such clients, and this adds the same opt-in contract. ## How checked - `python -m unittest serve.test_server serve.test_request_accounting serve.test_mcp` — includes 10 new tests: default unchanged, `include_usage`-only unchanged, monotonic cumulative usage on every data chunk (role chunk starts at 0), identical deltas/content/final usage with and without the option, monotonic under `max_tokens` truncation, two concurrent requests never cross, continuous usage across a reasoning-budget wrap-up keeps the existing accounting (original API prompt, `cached_tokens` == 5), two MCP rounds never reset the count (tool-running chunks carry the round total). - Live against a running server: streamed reasoning chunks carry incrementing cumulative `completion_tokens`; the final usage matches. - No C++/CUDA/HIP/Metal changes; no GPU needed for the tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.