Pull requests / #2928

#2928 feat(sycl): opt-in fdinfo-only sampler for Intel GPU load and VRAM

open · @pankleks · 0 comments · View on GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAWindowsLinux

Description

## Summary

Refs #2921. Companion to #2924, **not** a replacement for the server-side idle-telemetry fix. The two PRs use separate branches off `main` and share no changed files.

Intel GPU load and resident VRAM come from `/run/gpustat.json`. The supplied root sampler can provide them, but its full mode also continuously reads temperature, energy, clock and PCIe sensors. Installing that full sampler can therefore undo the benefit of stopping the server's own sensor reads after idle unload.

This adds **opt-in `GPUSTAT_FDINFO_ONLY=1`** to the existing sampler:

- Reuse its DRM fdinfo parser to publish selected-card load and resident VRAM to the same JSON file. No hwmon discovery/reads, power/temperature/fan/frequency sampling, PCIe-link reads or `lspci` call on this path. Card discovery and VRAM-capacity detection still read static sysfs metadata at import.
- Filter by `drm-pdev` before deduplicating DRM client IDs, so another card's clients cannot supply the selected card's numbers in this mode. Existing no-argument `clients()` behavior is unchanged.
- Report unknown load for first/reset/stalled intervals rather than a fabricated zero; report zero load/resident VRAM with no clients. Clamp the sum of client duty cycles at 100%, as the existing sampler does.
- Keep the default full hardware sampler intact. Add a documented systemd override; no automatic installation/enabling, new dependency, public API or engine change.
- Document the existing `GPUSTAT_VRAM_MB` override for SR-IOV/ReBAR cases. On the tested 32 GiB B70, the largest-resource heuristic selected a 56 GiB VF reservation; specifying 32768 MiB fixes the display. **This PR does not change that default heuristic.**

## Validation

- Windows: `python -m unittest tools.test_gpustat_fdinfo tools.test_setup_sycl` — **22 passed**.
- Linux: `python -m unittest tools.test_gpustat_fdinfo` — **4 passed**.
- Tests cover counter intervals/empty clients/clamping, a complete fdinfo-only publish with hardware helpers forbidden, unchanged default full-sampler fields, and per-card filtering before client-ID deduplication.
- Compared the original full-sampler `main()` AST with the new body after its opt-in dispatch: identical. No C++/CUDA/HIP/Metal source or build change.
- Candidate upstream code on the B70: executed its real fdinfo-only idle publish into a separate test file, verified no clients/zero resident VRAM and 32768 MiB configured total, and verified the existing Intel reader consumes that file. Production service/output were not replaced by this PR test.
- An **equivalent small local wrapper** using the same upstream fdinfo parser and load calculation is already running in an unprivileged Linux container on an Arc Pro B70, Strata 0.1.45 / Linux 7.0.14-24-pve. It restored live load (up to 100% during a 512-token-cap inference) and resident VRAM (~28.3 / 32 GiB). After unload, fresh no-client JSON and paused Monitor GPU fields were confirmed; passive PM counters with the wrapper alive showed +65,646 ms suspended / +9,359 ms active over ~75 seconds, starting/ending suspended. **Those live inference/PM numbers are from the local wrapper, not the exact new upstream mode.**

The sampler option alone does not stop the server's separate hardware reader; #2924 addresses that. No independent wattage reduction, uninterrupted suspend, exact-candidate inference/PM run, multi-GPU hardware run or engine rebuild is claimed. Ready for review as an Intel/xe-only, opt-in Python sampler change; the exact-candidate hardware validation limits above still apply. No private configuration or credentials included.

Related on this site

Quick links to related pages.