Pull requests / #3194

#3194 [Engine] serve: per-layer expert routing metrics for /metrics ("routing_counts", opt-in)

open · @blange48 · 0 Kommentare · Auf GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Beschreibung

## Problem and result

Which experts do the requests actually use? Today the only way to see it is a `--dump-routing` trace (~4 KB per generated token, rewritten at every start) and an offline script. This PR adds a cheap view of the same data on `/metrics`, so Prometheus and Grafana can show it.

- **Before:** no routing information on `/metrics`.
- **After, with the opt-in key:**
  - the JSON `/metrics` gets an `experts` section: per layer, the experts seen, the top 32's share, the entropy in bits and the top 10 experts;
  - the Prometheus text gets `strata:expert_routed_total`, the per-layer gauges `strata:expert_seen`, `strata:expert_top32_share` and `strata:expert_entropy_bits` (label `layer`), and `strata:expert_routed_by_expert_total` for each layer's 8 most routed experts (labels `layer`, `expert`). That is ~530 series, not 24,576.
- **Without the key** nothing is counted, written or reported.

## Related discussion

#1547, closed in the 0.2.0 tracker reset. The `/reopen` check passed there, but GitHub would not reopen it after the rebase force-push, so this is the same branch as a new pull request. The earlier review (SuperDistance, Windows builds included) is in #1547.

## Approach

- **Engine** (`--routing-counts PATH`, `--routing-counts-every MIN`): one `uint64` counter per (layer, expert), incremented where the routing trace reads the ids (`drive_pool` / `drive_pool_multi`, every layer of every decode window, drafted tokens included; prompt reads are not counted). Nothing is written per token. The counts are saved as JSON every minute between requests (the same spot as the expert-profile save) and on QUIT, through a temporary file renamed over the last one with `std::filesystem::rename`, so it replaces the old file on Windows too. A 48 × 512 file is ~55-85 KB.
- **Server** (`"routing_counts": "<path>"`, optional `"routing_counts_every"`): passes the flags, and `GET /metrics` reads the file, cached on its mtime.
- **Docs:** `docs/DETAILS.md`, and an Experts row in `docs/monitoring/grafana-strata.json`.
- Compatibility: off by default; no change to any other metric or to the engine's work when the key is absent.

## Validation

**Passed:**
- `python serve/test_prometheus.py` on this head (47e57d95, rebased on main): 8 tests OK (summary with counts, entropy, an empty layer, a missing file; gauges; per-expert counters, at most 8 per layer; config key to engine flags). macOS, Python 3.
- Engine and server built from v0.2.0-beta plus this PR (CUDA 13.0, `CUDA_ARCHITECTURES=120`, Linux, docker), then `pytest serve` in that image: 864 passed, 18 skipped, 1 failed: `test_lifeline.py::test_the_portable_watcher_ends_the_child_when_the_server_is_killed` (the POSIX watcher path inside a container; this PR does not touch it, I have not checked whether it fails on v0.2.0-beta alone).
- In production since 2026-10-08 on 0.1.40.3 to 0.1.45 builds: 4× RTX 5080, layer split 12,24,36, Qwen3.8-Flash-Next IQ3_S, `--batch 8 --batch-groups 4`. 384 `strata:expert_routed_by_expert_total` series exported at every release check, a Grafana row on top. Throughput with and without the key, 1 / 4 / 8 clients (T=0.7, 300 tokens, warm pass): 147 / 269 / 394 against 143 / 268 / 392 tok/s, within run-to-run noise. The counts agreed with a `--dump-routing` trace of the same traffic (layer 0's top experts 382, 300, 324 in both; every layer counted the same number of times).

**Failed:**
- none observed in this PR's code (the lifeline test above is outside it).

**Not run:**
- The runtime check on this exact head with 0.2.0-beta on the 4× RTX 5080 box: scheduled for tonight with our release gate; I will add the numbers here.
- Windows and HIP builds of this head (SuperDistance built an earlier revision on Windows for CUDA sm_120 and HIP gfx1100, which is how the rename bug was found and fixed).

## Limits

- Counts restart at zero with the engine (a Prometheus counter handles that). The first file appears at the first request a minute after the start.
- Prompt reads are not counted, only decode windows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.