Pull requests / #1933
#1933 serve: aggregate batch request timings
closed · @constantindjonkam · 0 Kommentare · Auf GitHub
Beschreibung
## What this fixes A request that moves through solo generation, batch admission, batch windows, and possibly back to solo was reporting only the final native segment. This produced inflated prompt rates and single-step decode rates for concurrent requests. This change: - accumulates prompt/decode clocks and native counters across all DONE/BDONE phases of one API request; - avoids double-counting the batch admission token when BDONE's generated count includes it; - accounts for a queued BDONE when a stop token/string closes the generator; - derives `timings.prompt_n` and Prometheus prompt totals from the engine's `prompt_read` field when available; - adds regression coverage for cumulative batch stats and slot-reused prompt timing. Fixes the behavior described in issues #1869 and #1870. ## Validation - `python -m py_compile serve/server.py` - `python -m unittest serve.test_server.BatchRequestStats serve.test_request_accounting -v` - `python -m unittest serve.test_parallel -v` - `python -m unittest discover -s serve -p 'test_*.py' -q` (692 tests; one existing order-sensitive sampler-thread test was flaky in the full run and passed in isolation)
Mehr auf der Site
Links zu Install, Modellen, Releases.