Issues / #2857
#2857 [Feature]: Monitor: one widget per batch slot, so several requests at once each show their own speed
open · @street-tripy · 0 comentários · No GitHub
BenchmarksServer & APIModels & quants
Descrição
When `"parallel": N` is set, I want to see at a glance that the requests are really decoding *together*, each with its own rate — and which slot is reading, decoding or free. Today the Monitor only shows one request's number: the header pill reads `Generating · 79.8 tok/s` (the newest request) and the Model state card shows one request's progress and rate. With two coding agents running I cannot tell from the page whether they are interleaved or one after the other, so I end up watching the engine log's `strata batch: ... avg 2.00 rows, ... ms/window` line instead — which is a strange thing to have to do to answer "is it busy?". What I would like: one widget per batch slot in the Monitor — `Worker 1`, `Worker 2`, … — each with the slot's state (Reading / Decoding / Free), its own live tok/s, how many tokens it has written and how long it has been in the slot, and that slot's context fill. The header pill would name every decoding stream when more than one is decoding (`2 decoding · 80.4 + 76.1 tok/s`). Workaround I use today: `curl /metrics` and read `live.slots` by hand, or the engine log. The data is already there — `live.slots` has carried `slot`, `state`, `prompt_tokens`, `generated`, `elapsed_s` and `tok_s` per slot since #465; the page simply does not show it. It should stay out of the way for everyone else: the card only appears when `/metrics` reports `live.parallel`, i.e. when the run config sets `"parallel": N`. Without it the Monitor is exactly what it is today. I have a patch ready for this (4 files, 98 insertions: `index.html`, `app.js`, `app.css`, plus the same page test as `test_the_card_is_on_the_page` in `test_monitor.py`) and will open a PR against it if that is welcome.
No site
Links install, modelos, releases.