Issues / #2857

#2857 [Feature]: Monitor: one widget per batch slot, so several requests at once each show their own speed

open · @street-tripy · 0 comentários · No GitHub

BenchmarksServer & APIModels & quants

Descrição

When `"parallel": N` is set, I want to see at a glance that the requests are really decoding
*together*, each with its own rate — and which slot is reading, decoding or free.

Today the Monitor only shows one request's number: the header pill reads `Generating · 79.8
tok/s` (the newest request) and the Model state card shows one request's progress and rate. With
two coding agents running I cannot tell from the page whether they are interleaved or one after
the other, so I end up watching the engine log's `strata batch: ... avg 2.00 rows, ... ms/window`
line instead — which is a strange thing to have to do to answer "is it busy?".

What I would like: one widget per batch slot in the Monitor — `Worker 1`, `Worker 2`, … — each
with the slot's state (Reading / Decoding / Free), its own live tok/s, how many tokens it has
written and how long it has been in the slot, and that slot's context fill. The header pill
would name every decoding stream when more than one is decoding (`2 decoding · 80.4 + 76.1
tok/s`).

Workaround I use today: `curl /metrics` and read `live.slots` by hand, or the engine log. The
data is already there — `live.slots` has carried `slot`, `state`, `prompt_tokens`, `generated`,
`elapsed_s` and `tok_s` per slot since #465; the page simply does not show it.

It should stay out of the way for everyone else: the card only appears when `/metrics` reports
`live.parallel`, i.e. when the run config sets `"parallel": N`. Without it the Monitor is
exactly what it is today.

I have a patch ready for this (4 files, 98 insertions: `index.html`, `app.js`, `app.css`, plus
the same page test as `test_the_card_is_on_the_page` in `test_monitor.py`) and will open a PR
against it if that is welcome.

No site

Links install, modelos, releases.