Issues / #457
#457 /metrics: expose the speculative draft counts (offered / accepted) for llm-visuals
closed · @blaakstone · 1 comentarios · En GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quants
Descripción
Hi, I run Strata 0.1.34 with Qwen3.8-Flash-Next IQ3_S at 262K on my RTX 5090 behind llama-swap (119-135 tok/s decode, ~5,000 tok/s prefill, 0.8 s to resume a 71K conversation), thank you for it.
I monitor it live with [llm-visuals](https://github.com/DingoOz/llm-visuals), a terminal dashboard that polls Strata's `/metrics` every second (its Strata adapter is on the branch `feature/strata`). Throughput, context, layers and requests work; the speculative decoding panel stays empty: "Strata does not report draft acceptance".
The server already has the MTP draft counts: `serve/server.py` reads them from the engine's DONE line (`self.last.update(drafts_accepted=..., drafts_offered=...)`), returns them in the response `timings` as `draft_n` / `draft_n_accepted` (#83), and the server log prints them ("drafts accepted 601 of 1181").
They are missing from `GET /metrics`: not in `live`, not in the `requests` history, not in `totals` (checked on 0.1.34). From the response `timings` I only get the figure once a request has finished; what I would like to see in llm-visuals is the acceptance while the model generates.
**Proposal** (additive, no change to existing fields):
- in `live`, while generating: `"drafts_offered"` and `"drafts_accepted"` so far for the current request (if the engine can report them before DONE);
- in each `requests` entry: the same two counts for the finished request (`null` when the engine did not report them, as `draft_n` is omitted today);
- in `totals`: running sums of both.
En el sitio
Enlaces a install, modelos, releases.