贡献 / #1593

#1593 serve: the hardware sampler keeps running where psutil finds no disk counters

open · @ischencheng · 0 评论 · 去 GitHub 看

Server & APIDocumentationWindowsLinux

说明

`psutil.disk_io_counters()` returns None when it finds no disk. On Modal (gVisor, no /proc/diskstats) that happens on every start, and psutil's docs say Windows can need `diskperf -y` before it sees one. `_disk()` then raises at the first sample and the sampler thread ends:

```
  File "serve/telemetry.py", line 311, in _disk
    prev, self._disk_prev = self._disk_prev, (t, c.read_bytes, c.write_bytes)
AttributeError: 'NoneType' object has no attribute 'read_bytes'
```

After that `/metrics` returns an empty `hardware`, the Monitor tab has nothing to show, and the frozen-engine check's `gpu_busy()` (#1317) always reads a GPU load of 0. hulkbig's #1583 mentions the same error in its test run.

The fix returns no disk reading in that case, like the `OSError` branch above it. The other readings go on as before.

Tested:

- New `TelemetryNoDisks` test in serve/test_server.py, with a fake psutil whose disk counters are None: it fails on main with the traceback above and passes here. serve/test_server.py (272 tests), test_monitor, test_prometheus, test_responses, test_lifecycle and test_parallel pass.
- Modal L4 24 GB, Linux (gVisor), engine 0.1.41 built from fb58e0d, IQ2_XS, the same config for both runs: one short request, then `/metrics`.

| | main | this branch |
|---|---|---|
| keys in `hardware` | 0 | 19 |
| samples in `history` | 0 | 9 |
| sampler traceback in the server log | yes | no |

The reply was the same in both runs. Not tested on Windows, and I did not try to trigger the frozen-engine check itself.

本站相关内容

相关页面的快捷入口。