Pull requests / #3492
#3492 [Engine] serve: a slot's live view keeps the request's own clocks after it moves from the solo path (elapsed_s, tok_s)
open · @andyhrc · 0 comentarios · En GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Descripción
## Problem and result With `--batch`, a request that starts alone on the solo path continues in a batch slot when another request arrives. The slot admission then carries the reply so far. Each admission restarted the slot row's clocks in `/metrics` `live.slots`, but `generated` kept counting the whole reply. As a result: - `elapsed_s` started again at 0. - `tok_s` was the whole reply divided by the time in the current leg only. It started thousands of times too high and fell as the leg went on. The test below, with a fake engine that writes one token per 20 ms (50 tok/s), shows it on current `main`. The row of the request that moved read: | generated | elapsed_s | tok_s | | ---: | ---: | ---: | | 19 | 0.0 | 3,945.9 | | 22 | 0.1 | 203.2 | | 27 | 0.3 | 127.1 | We first noticed it on our own monitor: right after a move, a request decoding at about 40 tok/s showed 200+ tok/s. After the change, the row uses the request's own start time and first token, as the server's per-request status already does. `elapsed_s` counts from the request's start, and `tok_s` is its rate since its first token, whichever leg it is in. ## Related discussion #2857 (closed in the 0.2.0-beta reset) asks for one Monitor widget per batch slot, showing that slot's own tok/s and time in the slot from `live.slots`. These are the fields that were wrong after a move. Nothing in `apps/` reads the per-slot `tok_s` / `elapsed_s` today; `/metrics` consumers that read `live.slots` (ours does) get the wrong numbers. ## Approach `generate_batched` keeps two clocks per request: when it began, and its first token, set by the token callback of whichever leg delivers it. The slot's live row takes `started` from the first and `first_token` from the second instead of `time.time()` at each admission. There is no other behaviour change: `slots_view` computes the same formula from these fields. ## Validation **Passed:** - **Revision.** `1625b9aa` on `main` `cbfd5b92` (v0.2.1.3), Ubuntu 24.04, Python 3.12 (`.venv`), AMD EPYC 7543. - **New test.** `python -m unittest serve.test_parallel_yield.ParallelServiceYield.test_a_continued_leg_shows_the_requests_own_clocks` passes. With the old `serve/server.py` it fails: `0.0 not greater than or equal to 0.3`, with the rows above. - **Serve suite.** `python -m unittest discover -s serve -t .`: 894 tests OK (18 skipped). - **The CI job's form** (a venv with only `requirements.txt`, then `tools/minimal_server_smoke.py` and `python -m unittest discover -s serve -p 'test_*.py'`): smoke OK, 894 tests OK (18 skipped). - **Hardware.** The same change runs on our server (3 x RTX A4000, Qwen3.8-Flash-Next IQ3_S, `--batch 6` in 3 groups; there it sits on top of our slot PRs #2527/#2528/#2843). A 700-token request ran alone for 3 s, then moved into slot 0 when a second request arrived. Its row read `elapsed_s` 3.4 -> 4.4 and `tok_s` 57.2 -> 50.9 while the second one decoded (2026-10-21, on 0.2.1.3). The request's own rate was 700 tokens in 12.5 s. **Failed:** None observed. **Not run:** a GPU run of `main` without the change. The "before" numbers are from the test above and our earlier observation. ## Limits The clocks are per request. A slot row shows the request's rate since its first token over all of its legs, not the slot's rate in the current window. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
En el sitio
Enlaces a install, modelos, releases.