Pull requests / #2890
#2890 serve: the Monitor shows the control-line wait a batch request is in
open · @dsj027-collab · 0 comentarios · En GitHub
Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux
Descripción
## What this changes With `--batch`, the server's fifo is no longer where requests queue: each one is registered as running the moment it arrives, and the queue it really waits in is the engine's control lines - one prompt read at a time, behind whatever read is in progress (`_take_control`). The Monitor could not show that anywhere: - the api-monitor's request list went `queued` -> `generating` in the same second and `queue_s` stayed `0.000`, so a request that waited 55 s for the next read looked identical to one that started reading at once; - the Monitor's pill only showed `N queued`, which with `--batch` is always 0 while a real queue is standing on the lines. `_take_control` now marks the request's own monitor record `waiting` while it is in the queue and `reading` when it takes the lines, and adds the seconds it spent waiting to that record's `queue_s` - the field the server's fifo wait already writes, so on `--batch 1` and `--batch N` alike the row shows the whole time before work started. The request list and the pill render the two new states (`N waiting to read` beside `N queued`); `/metrics` has carried `live.waiting`/`live.admitting` for a while now, this only draws them. ## What it leaves alone No scheduling change and no output change: the default path is byte-identical. Nothing is marked when `api_monitor` is off (the default) - there is no record to mark - and the `waiting` counter, `wait_lens` and `STRATA_ADMIT_SHORT_FIRST` admission order are untouched (a test asserts they still balance). ## How I checked - `serve/test_monitor_waiting.py` (new, 5 tests): the stubbed-engine pattern of `serve/test_admit_order.py` - no process, no GPU. Covers: free lines land on `reading` with a sub-second `queue_s`; held lines show `waiting` with nothing counted, then `reading` with the wait added; a cancelled wait never claims `reading`, keeps `queue_s` at 0, and leaves `waiting`/`wait_lens` balanced; no record on the thread is inert; the fifo's wait and the lines' wait add up in one field. - `serve.test_admit_order` + `serve.test_slot_alloc` (11 tests) and `serve.test_server` (337 tests) pass unchanged. - Live on an RTX PRO 6000 96 GB (Linux, CUDA sm_120, NVFP4 pack, `--batch 2`, engine 0.1.45): 2 x 120K-token requests plus 8 short ones at once - the list showed `waiting` rows with `queue_s` 54-58 s for the slot-bound requests and 0.9 s for the one `STRATA_ADMIT_SHORT_FIRST` let through, matching `/metrics` `live.waiting` (peak 8) sampled at 0.5 s. Not tested: Windows (the change is OS-independent Python + the web app), HIP/SYCL (no engine code touched). Follow-up to #2109 (its scheduling half shipped as `STRATA_ADMIT_SHORT_FIRST` in 0.1.45; this is the visibility half), and adjacent to #2857/#2858 (per-slot widgets).
En el sitio
Enlaces a install, modelos, releases.