Pull requests / #2423
#2423 serve: GET /metrics gives each request's PCIe expert bytes and its own KV streaming share, and a bandwidth summary
open · @MrKop-git · 0 コメント · GitHub で見る
本文
## Summary Observability for the bandwidth view (the link-tier work in #2099), server side only: - **Per request** — `offloaded` (the DONE line's count of routed experts the GPU read over PCIe) as `pcie_experts_mib` and `pcie_experts_gib_s` over the request's decode time. The blob size is the **mean the engine reports for its own expert cache** (its INFO line's `expert_cache_mib` over `expert_slots`) - a pack's blobs differ in size (a native pack has one size per layer: the default is 1,382,400 B, NVFP4 2.76 MB, IQ2_XS 1.51 MB), so no constant is assumed and the fields are `null` when the engine did not report enough. - **Per request** (KV streaming on) — `kv_block_reads`, `kv_hit_pct`, `kv_ram_mib`: one request's own share of the engine's per-request `KV streaming:` summary. That summary's counters are **cumulative over the process** (generate.cpp's own comment), so the server reports each line's difference from the previous one and matches it to the request that owns it - the line's own numbers are not a request's share. A request whose engine died before writing its summary keeps `null`; the fields are absent while KV streaming is off. - **`GET /metrics`** — a `"bandwidth"` object: mean and peak of the 1 s hardware series since the start (`gpu_pcie_rx_mb`, `gpu_pcie_tx_mb`, `disk_read_mb`, `disk_write_mb`; `null` without hardware telemetry). ## What it changes and what it leaves alone Additive fields in the JSON of `GET /metrics` (the same way `conversation_cache` was added), plus the fields' description in docs/DETAILS.md. **The engine, its DONE line and every answer are untouched** - no flag, no default change. ## How it was checked - `serve/test_server.py` (`PcieExpertsBytes`): the bytes and GiB/s from the engine's own cache size, and `null` for zero offloaded / no INFO / an older engine's DONE line. - `serve/test_monitor.py` (`KvStreamingLines`): two cumulative summaries become their differences (1000 then 2000 reads, 0.8 then 1.0 MiB, 80.0% then 87.5% per request), taken once each and attached to the right request rows through `GET /metrics`. - The full serve suite passes: **789 tests, 11 skipped**. - Not tested: a live capture against a real engine's `KV streaming:` line (the exact float formatting is pinned by the tests' strings against generate.cpp's `%.2f%% of %llu block reads` / `%.1f MiB`), and multi-slot (`--batch`) attribution - with several slots finishing at once the lines still arrive in finish order and the rows match that order. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
関連リンク
インストール・モデル・リリースへの站内リンク。