#6105 OpenAI streaming: opt-in continuous_usage_stats (vLLM-style cumulative usage on chunks)

open · @woniubility · 0 评论 · 去 GitHub 看

BenchmarksServer & APIAMD / HIPNVIDIA / CUDASecurity

说明

## What changes

- Adds opt-in `stream_options.continuous_usage_stats` to `/v1/chat/completions`: when `stream` + `include_usage` + `continuous_usage_stats` are all true, every data chunk carries the request's **cumulative** usage (`prompt_tokens` / `completion_tokens` / `total_tokens`), so clients that compute a live rate from streamed usage (e.g. a coding agent's thinking-speed badge) update while the model is still generating.
- The count is `Service._run`'s own `n` — thinking, tool-call argument tokens, wrap-up/close-retry tokens and every MCP round are included and never reset mid-request. It is not a text-length, SSE-chunk-count or re-tokenization estimate, so mid-stream values can never disagree with the final usage.
- A per-request `progress` dict is created only for requests that ask for the option, shared by the MCP run and `openai_chunks()`; the final chunk's authoritative usage (with accurate `prompt_tokens_details` and `timings`) overrides the mid-stream snapshot, which omits cache details it cannot know yet.

## What remains unchanged

- Default streams and final usage accounting (a request without `continuous_usage_stats` - or with only `include_usage` - is byte-identical to before, usage only on the final chunk, as today even without `include_usage`).
- Model output, sampling, detokenization, tool parsing and engine decode.
- Anthropic `/v1/messages` and Responses `/v1/responses`.

## Motivation

Clients like omp show a real-time tok/s badge for streaming thinking, computed as `usage` deltas between chunks. Today Strata only reports usage on the final chunk, so the badge stays hidden until generation ends; vLLM's `continuous_usage_stats` already feeds such clients, and this adds the same opt-in contract.

## How checked

- `python -m unittest serve.test_server serve.test_request_accounting serve.test_mcp` — includes 10 new tests: default unchanged, `include_usage`-only unchanged, monotonic cumulative usage on every data chunk (role chunk starts at 0), identical deltas/content/final usage with and without the option, monotonic under `max_tokens` truncation, two concurrent requests never cross, continuous usage across a reasoning-budget wrap-up keeps the existing accounting (original API prompt, `cached_tokens` == 5), two MCP rounds never reset the count (tool-running chunks carry the round total).
- Live against a running server: streamed reasoning chunks carry incrementing cumulative `completion_tokens`; the final usage matches.
- No C++/CUDA/HIP/Metal changes; no GPU needed for the tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。