Issues / #3157
#3157 Monitor: every GPU tile silently shows "-" when a Linux HIP config lacks "backend": "hip"
open · @destroyor · 0 comments · View on GitHub
BenchmarksSetup & installServer & APIAMD / HIPLinux
Description
## Monitor: every GPU tile silently shows "–" when a Linux HIP config lacks `"backend": "hip"`
**Box:** RX 7900 XTX (gfx1100), Arch Linux, engine 0.1.45 / server 0.1.45, HIP backend (ROCm 10.2 nightly). Full benchmark of this machine in #2938.
### Symptom
The web app's Monitor tab shows five empty tiles - GPU load, VRAM, GPU temp, Power, PCIe - while
`8,883 experts cached` underneath VRAM is fine (that subtitle comes from the engine's own report,
not from telemetry). About says `GPU: not readable (NVML)` on a box with no NVIDIA card, and
`hardware_static.gpu_note` is `None`: the dashboard gives no hint why, and the server log stays
silent. It looks like "this platform cannot read AMD GPUs", which is wrong - Linux can since #301.
### Root cause
The sysfs reader is gated on one config key, read once at server start:
- `serve/server.py`: `svc.backend = cfg.get("backend")`, then `Telemetry(..., amd=getattr(self, "backend", None) == "hip")`
- `serve/telemetry.py` `gpu_reader()`: `amd=False` on Linux means `_Nvml` - `libnvidia-ml.so.1` fails to load, `ok()` is False, and `sample()` skips the whole `gpu_*` block.
Two details make the failure fully silent:
1. the explanatory `gpu_note` is only produced when **`amd and not self.gpu.ok()`** - with the key
missing `amd` is False, so even the note path is skipped, and
2. `engine/BUILD.json` next to the exe already says `{"backend": "hip"}` (setup writes it, the HIP
packaging writes it), but nothing in `serve/` ever reads it, and there is no fallback probe.
A config written before #301 (or by hand / from an old guide) simply never gets the key; the box
then loses the whole Monitor tab without a word. Ours predates #301 and had been blank the entire time.
### Evidence
With no config change, the repo's own reader on the same box reads everything:
```text
$ .venv/bin/python -c "from serve.telemetry import gpu_reader; g=gpu_reader(0, amd=True); print(g.ok(), g.read())"
True {'util': 30, 'mem_used': 25351368704, 'mem_total': 25753026560, 'temp': 47.0, 'power': 88.0,
'power_limit': 339.0, 'pcie_gen': 4, 'pcie_gen_max': 4, 'pcie_own_gen': 4, 'pcie_width': 16, 'pcie_path': '4/4'}
```
Adding `"backend": "hip"` to the config and restarting brought all five tiles back, live and
correct (util 0 -> 95 % under a request, temp 37 -> 43 C, power 7 -> 247 W, VRAM 23.5 GB, Gen4 x16).
### Suggested fixes (any one would have saved an hour of source-diving)
1. **Derive the backend when the key is absent**: when `cfg` has no `"backend"` and the exe has a
sibling `BUILD.json`, use its `backend` value - the information is already stamped at install time.
2. Or a runtime fallback in `gpu_reader()`: on Linux, when NVML fails to load and
`/sys/class/kfd/kfd/topology/nodes` lists an AMD GPU, use `_Amd` (it needs no ROCm libraries).
3. At minimum, widen the note: when on Linux, NVML failed, no `backend` is set and the KFD shows an
AMD card, set `gpu_note` to say the config is missing `"backend": "hip"` instead of staying None.
(The PCIe tile showing Gen/width but no MB/s on AMD is separate and already covered by closed #1734.)
### Workaround
Add `"backend": "hip"` to the serve config JSON and restart the server. It also re-enables the
adjacent `backend == "hip"` features that were silently off: the free-VRAM admission check
("then nothing is refused"), the desktop-VRAM note (#560/#516) and the GPU-busy exemption of the
frozen-engine watchdog (#1317).
Related on this site
Quick links to related pages.