Pull requests / #1933

#1933 serve: aggregate batch request timings

closed · @constantindjonkam · 0 comentarios · En GitHub

Server & APIWindows

Descripción

## What this fixes

A request that moves through solo generation, batch admission, batch windows, and possibly back to solo was reporting only the final native segment. This produced inflated prompt rates and single-step decode rates for concurrent requests.

This change:

- accumulates prompt/decode clocks and native counters across all DONE/BDONE phases of one API request;
- avoids double-counting the batch admission token when BDONE's generated count includes it;
- accounts for a queued BDONE when a stop token/string closes the generator;
- derives `timings.prompt_n` and Prometheus prompt totals from the engine's `prompt_read` field when available;
- adds regression coverage for cumulative batch stats and slot-reused prompt timing.

Fixes the behavior described in issues #1869 and #1870.

## Validation

- `python -m py_compile serve/server.py`
- `python -m unittest serve.test_server.BatchRequestStats serve.test_request_accounting -v`
- `python -m unittest serve.test_parallel -v`
- `python -m unittest discover -s serve -p 'test_*.py' -q` (692 tests; one existing order-sensitive sampler-thread test was flaky in the full run and passed in isolation)

En el sitio

Enlaces a install, modelos, releases.