Pull requests / #2359

#2359 serve: average the throughput gauges over the window between scrapes (#2341)

open · @nekomario28 · 0 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIModels & quants

Beschreibung

Closes #2341.

`/metrics/prometheus` exposed `llamacpp:prompt_tokens_seconds` and `llamacpp:predicted_tokens_seconds` as `total_tokens / total_seconds` over the server's lifetime, so a server idle for hours still advertised e.g. 58 tok/s, with nothing in the text format to say how old the number was.

llama.cpp does not do that under those names. Both rates come from `server_metrics::bucket` (`tools/server/server-common.h`), which a scrape resets — *"the gauges are averaged over the window between two scrapes"* (`tools/server/server-context.cpp`) — and `bucket::n_per_second()` returns `0.0` when the window is empty. So the same dashboard that decays to `0` against llama.cpp froze at the last request's speed against Strata.

The `*_total` counters stay cumulative; only the two gauges now come from a bucket held across scrapes, exposed as the delta. The HELP text says which window they cover.

## Testing

`python -m unittest serve.test_server.PrometheusMetrics`

The new case sends one request, scrapes, then scrapes again with no traffic in between and requires `(0, 0)` for the two gauges while `llamacpp:tokens_predicted_total` stays put. Against the old code that second scrape returns the frozen `(602.5, 50.0)` (the failure is in the report), so the case is a real guard rather than a restatement.

`python -m unittest discover -s serve -t . -p 'test_*.py'` — 770 tests, `OK (skipped=8)`, with this checkout's `requirements.txt` installed (a bare system Python was missing `regex`, which is unrelated to the change).

Mehr auf der Site

Links zu Install, Modellen, Releases.