改动 / #3117

#3117 gpustat: the Intel VRAM readings on a host with two cards

open · @beyondcr · 0 评论 · 去 GitHub 看

Setup & installServer & APIAMD / HIPWindows

说明

## Setup

Two Intel GPUs on one xe host, both behind the chipset's PCIe bridges:

- Arc Pro B70, `0000:0d:00.0` (Strata runs here)
- Arc A770, `0000:11:00.0` (drives the desktop)

Resizable BAR is enabled on both, so each card maps all of its VRAM through BAR2. Driver and
kernel are current, and both cards report correctly in `lspci`, in Level Zero and in `eglinfo`,
so the hardware side is not in question.

## What we saw

The Monitor's VRAM card read **31.2 of 56.0 GB** for the card Strata was running on, which has
30.30 GiB of usable VRAM. The total was almost double the card. A second sample taken with
nothing loaded still read 56.0 GB, so it was not a transient reading.

Correcting the total alone then made the panel read **105 percent**, which is how the second
problem surfaced.

## Cause 1: the total was a bridge window

`vram_total_mb()` took `max()` over every line of
`/sys/bus/pci/devices/<PCI>/resource`:

```python
sizes = [int(e, 16) - int(s, 16) + 1 for s, e, _ in
         (l.split() for l in open(f"/sys/bus/pci/devices/{PCI}/resource") if l.strip())]
return max(sizes) // 2**20
```

That file holds thirteen lines: the card's six BARs and its ROM at indices 0-6, then the bridge
windows the card sits behind at 7-12. A bridge window is sized for everything behind it, so it
can be much larger than the card. On our host, above 1 GiB:

```
index 0   BAR      0.02 GiB
index 2   BAR     32.00 GiB    <- BAR2, maps all of VRAM with resizable BAR
index 7   bridge   0.11 GiB
index 9   bridge  56.00 GiB    <- what max() returned
```

The largest line was a bridge window, so a bridge window became the card's capacity.

The module already documents what the value should be: *"the card's largest PCI BAR, which maps
all of VRAM with resizable BAR - a little above what the runtime reports as usable"*. A bridge
window is none of those three things, so this makes the code agree with the documented intent.

## Cause 2: the used figure summed every card

`clients()` iterated `/proc/[pid]/fd` and accepted any path under `/dev/dri`, then deduplicated
by `drm-client-id`. It never looked at `drm-pdev`, so with two cards fitted the `used` figure was
both cards summed while `total` was one card.

Measured on our host, before the fix:

```
B70 (engine)      32161 MiB
A770 (display)     2352 MiB
sampler "used"    34514 MiB   <- both cards, against a 32768 MiB total
```

That is why the panel read 105 percent once the total was correct. The inflated 56 GiB total had
been masking it.

## The fix

Two changes in the same file:

```python
# the card's own BARs and ROM only
bar = [l.split() for l in open(f"/sys/bus/pci/devices/{PCI}/resource") if l.strip()][:7]
sizes = [int(e, 16) - int(s, 16) + 1 for s, e, _ in bar]
return max(sizes) // 2**20
```

```python
# only the sampled card's clients
pdev = info.get("drm-pdev")
if pdev and pdev != PCI:        # another card's client: its VRAM is not this card's
    continue
```

A client that reports no `drm-pdev` at all is still counted, so a kernel that does not publish
it keeps the previous behaviour rather than reporting nothing.

The total is still the BAR window rather than usable VRAM, so it rounds up to the next power of
two: 32 GiB for 30.30 GiB usable, 16 GiB for 15.11 GiB. That is the docstring's "a little above
what the runtime reports as usable", and this change does not alter it. `GPUSTAT_VRAM_MB` still
overrides everything.

Neither change is specific to a card model. Both apply to any Intel GPU whose host fits a second
card, which is a property of the platform rather than of the model or the amount of VRAM.

## Effect on our host

| reading | before | after |
|---|---|---|
| engine card total | 57344 MiB | **32768 MiB** |
| engine card used | 34499 MiB (both cards) | **32161 MiB** (its own) |
| engine card ratio | 60 percent | **98 percent** |
| display card used / total | 2376 / 16384 | 2376 / 16384 (14 percent) |

The display card was already correct on the total, because its VRAM BAR happened to be the
largest line.

## Tests

New `tools/test_gpustat_vram_total.py`, five CPU-only cases:

- a card whose bridge window is larger than its BAR, asserting the card's own BAR. **Fails
  against the previous implementation**, which returns the bridge window.
- the ordinary shape, where the VRAM BAR is the largest of the card's own regions.
- `GPUSTAT_VRAM_MB` still taking precedence.
- a client on another card, asserting it is not counted. **Fails against the previous
  implementation**, which returns `['a', 'b']` instead of `['a']`.
- a client that reports no `drm-pdev`, asserting it is still counted.

Both regression cases were run against the unfixed sampler to confirm they fail.

```
$ python3 -m unittest tools.test_gpustat_vram_total -v
test_a_bridge_window_larger_than_the_card_is_not_the_capacity ... ok
test_the_card_bar_is_used_when_no_bridge_window_is_larger ... ok
test_the_explicit_override_still_wins ... ok
test_another_cards_client_is_not_counted ... ok
test_a_client_without_a_pdev_is_still_counted ... ok

Ran 5 tests in 0.003s
OK
```

Verified end to end on the real sampler, as root, against both cards: 32161 of 32768 MiB and
2376 of 16384, with busy percent, temperature, power and the PCIe link all still reporting.

## Notes

- `serve/telemetry.py` and the sampler's JSON shape are untouched. Only how the two numbers are
  computed, plus the tests.
- Busy percent also becomes per-card as a side effect of the client filter, which is what it
  should have been.
- `#2001` is related but different: that report is about the second card not appearing at all.
  This does not address it.
- Written with AI assistance, then measured and reviewed by hand on the hardware above. I can run
  further checks on request; this machine has both cards fitted.

本站相关内容

相关页面的快捷入口。