Pull requests / #2272

#2272 Monitor: the PCIe traffic from the bytes the engine moved, not from a driver counter

open · @unimo-chan · 0 comments · View on GitHub

Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux

Description

## What this changes

The Monitor's PCIe tile shows a traffic rate only when the driver reports one. `_Nvml.read()` fills `pcie_rx_mb`
from `nvmlDeviceGetPcieThroughput`; nothing else does. An AMD card has no such counter to read: on Windows there
is no NVML, the performance-counter set has no PCIe entry (only the five GPU sets and RDMA Activity) and neither
`amdadlx64.dll` nor `atiadlxx.dll` exports a bus metric; on Linux amdgpu's `pcie_bw` counts messages, not bytes
(#1734). The one component that does know the traffic is the engine: it staged those expert blobs itself.

So the engine counts the bytes and the server turns them into a rate:

- `ExpertDispatch::pcie_bytes` (`include/strata/core/expert_source.hpp`) — one blob per staged expert, in every
  `--pcie-mode`: the copy engine writes them into staging, kernel and direct modes read the host's blob where it
  lies. `offload_entries` already counted routed *entries*, not bytes. Counted in
  `expert_pool_dispatch_multi`, the same place `pcie_experts` is.
- The engine prints the running total on `PCIE <bytes>` lines at the points where it already flushes a token line
  (both decode loops and both batch loops), and adds one field to `DONE`: the request's own MB.
- `serve/server.py` keeps `(time, bytes)` samples and takes the slope over the same window the token rate uses
  (`RATE_WINDOW_S`), and reads the request's figure from the `DONE` line. It fills `gpu_pcie_rx_mb` **only when
  the reader says it has no traffic counter of its own** — a `has_pcie_throughput` flag, True on NVML, False on
  the AMD paths — so an NVIDIA card keeps showing NVML's own numbers.
- The tile puts the traffic next to the bandwidth the engine measured at its own start (`PCIe probe: N GB/s
  host->device`, already on stderr, which the server owns as the log): `412 MB/s · 12.1% of 3.4 GB/s measured`.
  The request list's `+X% PCIe` badge gains the MB behind it.

## What it leaves alone

- The generation half of the tile: no `pcie_gen`/`pcie_width` is added anywhere, and nothing turns a measured
  GB/s into a generation (Gen4 x8 and Gen3 x16 measure the same). #2137's point about the Gen headline is a
  separate change.
- NVML's readings, and amdgpu's `pcie_bw` — #1734's question about the three sysfs integers is untouched here;
  this answers the rate half from the engine instead, which works on Windows too.
- No new option and no default change: the engine's output gains lines, and the server only reads them. An engine
  older than this prints no `PCIE` lines and the tile stays as it is.

## How I checked

- `python -m unittest serve.test_server.DraftCounts serve.test_server.PrometheusMetrics serve.test_server.PcieShare
  serve.test_server.CancelledRead serve.test_server.LiveRate serve.test_server.WebApp serve.test_server.AmdTelemetry
  serve.test_server.TelemetryNoDisks` on this branch (main at `a0116b2`, Python 3.12.10): **27 tests, OK** — the
  classes that parse the engine's `DONE` line, serve `/metrics`, and run the sampler, with the stock tests
  unchanged. The `DONE` parser still accepts the shorter lines of an older engine: those tests pass as they are.
  So the `PCIE` lines and the new `DONE` field are exercised against the mock engine, not a real one. Full `serve.test_server` on this branch: **308 tests, OK** in 101 s (Windows 11, Python 3.12.10), nothing skipped.
- **Not built.** This machine has no CMake, MSVC, CUDA or SYCL toolchain, so the C++ side is not compiled here —
  the CUDA and SYCL trees get the same lines, and both are unverified by a compiler. A build of `src/` (CUDA/HIP)
  and `sycl/` before review is what AGENTS.md asks for; I could not do it, and I am saying so rather than
  implying it.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.