#6062 [Engine] Prometheus /metrics: request token histograms, finished_reason, conversation-cache gauges, telemetry key
open · @TokenGTx · 3 Kommentare · Auf GitHub
Server & APINVIDIA / CUDAModels & quantsSecurityLinux
Beschreibung
**Title:** `[Engine] Prometheus /metrics: request token histograms, finished_reason, conversation-cache gauges, telemetry key` ### Area [Engine] (speed, kernels, models, the server and API) ### What do you want to do? #### Background We run Strata next to vLLM on the same hosts, behind one gateway, and scrape both into VictoriaMetrics/Grafana with one scrape job. Strata's Prometheus mode on `/metrics` (vLLM metric names when the scraper sends a Prometheus `Accept` header) is excellent: the token-throughput, TTFT, ITL, e2e-latency and prefix-cache panels we built for vLLM picked up Strata with no dashboard changes. Thank you for that design choice. A few gaps remain, where the data already exists in Strata's JSON `/metrics` but is missing from the Prometheus text. In priority order: #### 1. Per-request token-count histograms Please add the vLLM-named histograms: - `vllm:request_prompt_tokens` (`_bucket` / `_sum` / `_count`): prompt tokens per finished request - `vllm:request_generation_tokens`: generated tokens per finished request Use case: a request-length heatmap (`sum by (le) (increase(vllm:request_prompt_tokens_bucket[$__rate_interval]))`). This is the main panel that currently shows nothing for Strata. The per-request numbers are already recorded (`requests[].prompt_tokens`, `requests[].output_tokens` in the JSON). vLLM uses 1-2-5 buckets up to the max model length. With Strata's 256K contexts, a bucket set reaching `262144` (and `+Inf`) would be ideal. #### 2. `finished_reason` label on `vllm:request_success_total` vLLM labels this counter with `finished_reason` (`stop`, `length`, `abort`, …). Strata's counter has no label, so success-rate panels (`finished_reason=~"stop|length"` ÷ all) and finish-reason pie charts are empty for Strata. The value is already known per request (`requests[].finish`). #### 3. Conversation-cache (parking) gauges and counters For hybrid models (e.g. Qwen3.8-Flash-Next) the conversation cache *is* the prefix cache. We want to see how close it is to evicting parked conversations, i.e. whether `conversation-cache-mib` / `conversation-cache-slots` are sized right. The JSON `conversation_cache` object has everything. Suggested names: | Metric | Type | From JSON | |---|---|---| | `strata:conversation_cache_budget_bytes` | gauge | `conversation_cache.budget_mib` × 2²⁰ | | `strata:conversation_cache_used_bytes` | gauge | `conversation_cache.bytes` | | `strata:conversation_cache_slots` | gauge | `conversation_cache.slots` | | `strata:conversation_cache_parked` | gauge | `conversation_cache.parked` | | `strata:conversation_cache_evictions_total` | counter | `conversation_cache.evictions` | | `strata:conversation_cache_parks_total` | counter | `conversation_cache.parks` | | `strata:conversation_cache_restores_total` | counter | `conversation_cache.restores` | | `strata:conversation_cache_reused_tokens_total` | counter | `conversation_cache.reused_tokens` | If the checkpoints of the live session have a separate budget, gauges for that tier would also help. The same goes for any VRAM-resident portion (`engine.kv_resident`) versus RAM, so the two tiers can be graphed separately. #### 4. Accept the telemetry read key on `/metrics` (Prometheus mode) Today `STRATA_TELEMETRY_KEY` / `telemetry_read_key` is honoured only on `/v1/telemetry`. A scraper of `/metrics` therefore needs the full inference API key, and that key ends up in every monitoring agent's config. Accepting the read-only telemetry key on `GET /metrics` (both formats, or at least the Prometheus one) would let monitoring hold a key that cannot run inference. That is what the telemetry key looks designed for. #### 5. (Nice to have) Stage-timing histograms For parity with vLLM's per-request breakdown panels: - `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` (from `requests[].prompt_ms`, `decode_ms`) - `vllm:request_time_per_output_token_seconds` - `vllm:request_queue_time_seconds` - `vllm:request_prefill_kv_computed_tokens`: prompt tokens actually computed, i.e. not reused (`prompt_read`) The totals counters (`strata:totals_prompt_seconds_total`, `strata:totals_decode_seconds_total`) let us derive averages already, so this is lower priority than 1–4. ### What do you do today instead? - **Request-length heatmap, success rate, finish reasons:** nothing. Strata shows up as missing on those panels, while the vLLM-served models on the same dashboards have them. - **Conversation cache:** we read it by hand from the JSON `/metrics` (`conversation_cache.*`) or the Monitor tab. Graphing it would mean writing and running our own JSON-to-Prometheus exporter, and we would rather not maintain that next to Strata. - **Auth:** our scraper is configured with the full inference API key, sent as `Authorization: Bearer` on every scrape. - **Stage timings:** per-request averages derived from `strata:totals_prompt_seconds_total` / `strata:totals_decode_seconds_total` ÷ `vllm:request_success_total`. #### Environment Strata 0.2.7 / engine 0.1.52, Docker image `ghcr.io/niko1221/strata:0.2.7-cuda13`. Linux x86_64, single RTX 3090, Qwen3.8-Flash-Next (IQ3_S and UD-Q4_K_XL), `PARALLEL=1`, scraped every 15 s by VictoriaMetrics `vmagent` v1.93.
Mehr auf der Site
Links zu Install, Modellen, Releases.