Pull requests / #3194
#3194 [Engine] serve: per-layer expert routing metrics for /metrics ("routing_counts", opt-in)
open · @blange48 · 0 comments · View on GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
## Problem and result Which experts do the requests actually use? Today the only way to see it is a `--dump-routing` trace (~4 KB per generated token, rewritten at every start) and an offline script. This PR adds a cheap view of the same data on `/metrics`, so Prometheus and Grafana can show it. - **Before:** no routing information on `/metrics`. - **After, with the opt-in key:** - the JSON `/metrics` gets an `experts` section: per layer, the experts seen, the top 32's share, the entropy in bits and the top 10 experts; - the Prometheus text gets `strata:expert_routed_total`, the per-layer gauges `strata:expert_seen`, `strata:expert_top32_share` and `strata:expert_entropy_bits` (label `layer`), and `strata:expert_routed_by_expert_total` for each layer's 8 most routed experts (labels `layer`, `expert`). That is ~530 series, not 24,576. - **Without the key** nothing is counted, written or reported. ## Related discussion #1547, closed in the 0.2.0 tracker reset. The `/reopen` check passed there, but GitHub would not reopen it after the rebase force-push, so this is the same branch as a new pull request. The earlier review (SuperDistance, Windows builds included) is in #1547. ## Approach - **Engine** (`--routing-counts PATH`, `--routing-counts-every MIN`): one `uint64` counter per (layer, expert), incremented where the routing trace reads the ids (`drive_pool` / `drive_pool_multi`, every layer of every decode window, drafted tokens included; prompt reads are not counted). Nothing is written per token. The counts are saved as JSON every minute between requests (the same spot as the expert-profile save) and on QUIT, through a temporary file renamed over the last one with `std::filesystem::rename`, so it replaces the old file on Windows too. A 48 × 512 file is ~55-85 KB. - **Server** (`"routing_counts": "<path>"`, optional `"routing_counts_every"`): passes the flags, and `GET /metrics` reads the file, cached on its mtime. - **Docs:** `docs/DETAILS.md`, and an Experts row in `docs/monitoring/grafana-strata.json`. - Compatibility: off by default; no change to any other metric or to the engine's work when the key is absent. ## Validation **Passed:** - `python serve/test_prometheus.py` on this head (47e57d95, rebased on main): 8 tests OK (summary with counts, entropy, an empty layer, a missing file; gauges; per-expert counters, at most 8 per layer; config key to engine flags). macOS, Python 3. - Engine and server built from v0.2.0-beta plus this PR (CUDA 13.0, `CUDA_ARCHITECTURES=120`, Linux, docker), then `pytest serve` in that image: 864 passed, 18 skipped, 1 failed: `test_lifeline.py::test_the_portable_watcher_ends_the_child_when_the_server_is_killed` (the POSIX watcher path inside a container; this PR does not touch it, I have not checked whether it fails on v0.2.0-beta alone). - In production since 2026-10-08 on 0.1.40.3 to 0.1.45 builds: 4× RTX 5080, layer split 12,24,36, Qwen3.8-Flash-Next IQ3_S, `--batch 8 --batch-groups 4`. 384 `strata:expert_routed_by_expert_total` series exported at every release check, a Grafana row on top. Throughput with and without the key, 1 / 4 / 8 clients (T=0.7, 300 tokens, warm pass): 147 / 269 / 394 against 143 / 268 / 392 tok/s, within run-to-run noise. The counts agreed with a `--dump-routing` trace of the same traffic (layer 0's top experts 382, 300, 324 in both; every layer counted the same number of times). **Failed:** - none observed in this PR's code (the lifeline test above is outside it). **Not run:** - The runtime check on this exact head with 0.2.0-beta on the 4× RTX 5080 box: scheduled for tonight with our release gate; I will add the numbers here. - Windows and HIP builds of this head (SuperDistance built an earlier revision on Windows for CUDA sm_120 and HIP gfx1100, which is how the rename bug was found and fixed). ## Limits - Counts restart at zero with the engine (a Prometheus counter handles that). The first file appears at the first request a minute after the start. - Prompt reads are not counted, only decode windows. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.