Pull requests / #2423

#2423 serve: GET /metrics gives each request's PCIe expert bytes and its own KV streaming share, and a bandwidth summary

open · @MrKop-git · 0 Kommentare · Auf GitHub

Server & APIDocumentation

Beschreibung

## Summary

Observability for the bandwidth view (the link-tier work in #2099), server side only:

- **Per request** — `offloaded` (the DONE line's count of routed experts the GPU read over PCIe) as `pcie_experts_mib` and `pcie_experts_gib_s` over the request's decode time. The blob size is the **mean the engine reports for its own expert cache** (its INFO line's `expert_cache_mib` over `expert_slots`) - a pack's blobs differ in size (a native pack has one size per layer: the default is 1,382,400 B, NVFP4 2.76 MB, IQ2_XS 1.51 MB), so no constant is assumed and the fields are `null` when the engine did not report enough.
- **Per request** (KV streaming on) — `kv_block_reads`, `kv_hit_pct`, `kv_ram_mib`: one request's own share of the engine's per-request `KV streaming:` summary. That summary's counters are **cumulative over the process** (generate.cpp's own comment), so the server reports each line's difference from the previous one and matches it to the request that owns it - the line's own numbers are not a request's share. A request whose engine died before writing its summary keeps `null`; the fields are absent while KV streaming is off.
- **`GET /metrics`** — a `"bandwidth"` object: mean and peak of the 1 s hardware series since the start (`gpu_pcie_rx_mb`, `gpu_pcie_tx_mb`, `disk_read_mb`, `disk_write_mb`; `null` without hardware telemetry).

## What it changes and what it leaves alone

Additive fields in the JSON of `GET /metrics` (the same way `conversation_cache` was added), plus the fields' description in docs/DETAILS.md. **The engine, its DONE line and every answer are untouched** - no flag, no default change.

## How it was checked

- `serve/test_server.py` (`PcieExpertsBytes`): the bytes and GiB/s from the engine's own cache size, and `null` for zero offloaded / no INFO / an older engine's DONE line.
- `serve/test_monitor.py` (`KvStreamingLines`): two cumulative summaries become their differences (1000 then 2000 reads, 0.8 then 1.0 MiB, 80.0% then 87.5% per request), taken once each and attached to the right request rows through `GET /metrics`.
- The full serve suite passes: **789 tests, 11 skipped**.
- Not tested: a live capture against a real engine's `KV streaming:` line (the exact float formatting is pinned by the tests' strings against generate.cpp's `%.2f%% of %llu block reads` / `%.1f MiB`), and multi-slot (`--batch`) attribution - with several slots finishing at once the lines still arrive in finish order and the rows match that order.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.