#6062 [Engine] Prometheus /metrics: request token histograms, finished_reason, conversation-cache gauges, telemetry key

open · @TokenGTx · 3 commentaires · Sur GitHub

Server & APINVIDIA / CUDAModels & quantsSecurityLinux

Description

**Title:** `[Engine] Prometheus /metrics: request token histograms, finished_reason, conversation-cache gauges, telemetry key`

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

#### Background

We run Strata next to vLLM on the same hosts, behind one gateway, and scrape both into
VictoriaMetrics/Grafana with one scrape job. Strata's Prometheus mode on `/metrics` (vLLM metric names
when the scraper sends a Prometheus `Accept` header) is excellent: the token-throughput, TTFT, ITL,
e2e-latency and prefix-cache panels we built for vLLM picked up Strata with no dashboard changes.
Thank you for that design choice.

A few gaps remain, where the data already exists in Strata's JSON `/metrics` but is missing from the
Prometheus text. In priority order:

#### 1. Per-request token-count histograms

Please add the vLLM-named histograms:

- `vllm:request_prompt_tokens` (`_bucket` / `_sum` / `_count`): prompt tokens per finished request
- `vllm:request_generation_tokens`: generated tokens per finished request

Use case: a request-length heatmap (`sum by (le) (increase(vllm:request_prompt_tokens_bucket[$__rate_interval]))`).
This is the main panel that currently shows nothing for Strata. The per-request numbers are already
recorded (`requests[].prompt_tokens`, `requests[].output_tokens` in the JSON). vLLM uses 1-2-5 buckets
up to the max model length. With Strata's 256K contexts, a bucket set reaching `262144` (and
`+Inf`) would be ideal.

#### 2. `finished_reason` label on `vllm:request_success_total`

vLLM labels this counter with `finished_reason` (`stop`, `length`, `abort`, …). Strata's counter has
no label, so success-rate panels (`finished_reason=~"stop|length"` ÷ all) and finish-reason pie
charts are empty for Strata. The value is already known per request (`requests[].finish`).

#### 3. Conversation-cache (parking) gauges and counters

For hybrid models (e.g. Qwen3.8-Flash-Next) the conversation cache *is* the prefix cache. We want to
see how close it is to evicting parked conversations, i.e. whether `conversation-cache-mib` /
`conversation-cache-slots` are sized right. The JSON `conversation_cache` object has everything.
Suggested names:

| Metric | Type | From JSON |
|---|---|---|
| `strata:conversation_cache_budget_bytes` | gauge | `conversation_cache.budget_mib` × 2²⁰ |
| `strata:conversation_cache_used_bytes` | gauge | `conversation_cache.bytes` |
| `strata:conversation_cache_slots` | gauge | `conversation_cache.slots` |
| `strata:conversation_cache_parked` | gauge | `conversation_cache.parked` |
| `strata:conversation_cache_evictions_total` | counter | `conversation_cache.evictions` |
| `strata:conversation_cache_parks_total` | counter | `conversation_cache.parks` |
| `strata:conversation_cache_restores_total` | counter | `conversation_cache.restores` |
| `strata:conversation_cache_reused_tokens_total` | counter | `conversation_cache.reused_tokens` |

If the checkpoints of the live session have a separate budget, gauges for that tier would also help.
The same goes for any VRAM-resident portion (`engine.kv_resident`) versus RAM, so the two tiers can
be graphed separately.

#### 4. Accept the telemetry read key on `/metrics` (Prometheus mode)

Today `STRATA_TELEMETRY_KEY` / `telemetry_read_key` is honoured only on `/v1/telemetry`. A scraper of
`/metrics` therefore needs the full inference API key, and that key ends up in every monitoring
agent's config. Accepting the read-only telemetry key on `GET /metrics` (both formats, or at least
the Prometheus one) would let monitoring hold a key that cannot run inference. That is what the
telemetry key looks designed for.

#### 5. (Nice to have) Stage-timing histograms

For parity with vLLM's per-request breakdown panels:

- `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` (from `requests[].prompt_ms`, `decode_ms`)
- `vllm:request_time_per_output_token_seconds`
- `vllm:request_queue_time_seconds`
- `vllm:request_prefill_kv_computed_tokens`: prompt tokens actually computed, i.e. not reused (`prompt_read`)

The totals counters (`strata:totals_prompt_seconds_total`, `strata:totals_decode_seconds_total`) let us
derive averages already, so this is lower priority than 1–4.

### What do you do today instead?

- **Request-length heatmap, success rate, finish reasons:** nothing. Strata shows up as missing on
  those panels, while the vLLM-served models on the same dashboards have them.
- **Conversation cache:** we read it by hand from the JSON `/metrics` (`conversation_cache.*`) or the
  Monitor tab. Graphing it would mean writing and running our own JSON-to-Prometheus exporter, and we
  would rather not maintain that next to Strata.
- **Auth:** our scraper is configured with the full inference API key, sent as `Authorization: Bearer`
  on every scrape.
- **Stage timings:** per-request averages derived from `strata:totals_prompt_seconds_total` /
  `strata:totals_decode_seconds_total` ÷ `vllm:request_success_total`.

#### Environment

Strata 0.2.7 / engine 0.1.52, Docker image `ghcr.io/niko1221/strata:0.2.7-cuda13`. Linux x86_64,
single RTX 3090, Qwen3.8-Flash-Next (IQ3_S and UD-Q4_K_XL), `PARALLEL=1`, scraped every 15 s by
VictoriaMetrics `vmagent` v1.93.

Sur le site

Liens install, modèles, releases.