#5602 [Engine] tests: test_hip_ordinal fails on any machine with an integrated Radeon

closed · @rwkeyes · 0 comments · View on GitHub

Server & APIAMD / HIPWindowsLinux

Description

## Problem and result

`serve.test_server.GpuChoice.test_hip_ordinal` fails on every machine that has an integrated Radeon — and passes on
CI, where no AMD card can be probed. That is why it has been red in the release for everyone running Strata on an AMD
box and green upstream.

The failing assertion:

```python
self.assertEqual(child_env({"backend": "hip", "gpu": [1, 0], "hip_ordinal": 2})["HIP_VISIBLE_DEVICES"],
                 "1,0")                                     # a layer split keeps its list
```

…gets `"0,1"`. `child_env` → `hip_visible` → `ordered_gpus`, which **deliberately** moves built-in graphics first
(#1576: the last stage runs the head, the draft layer and the verification, and an iGPU usually draws the desktop), and
`hip_gpu_archs()` reads the host's KFD topology to know which card that is. So the test asserts the config's order
unconditionally while the code consults the machine — a property of the test, not of the engine.

**Reproduced on the v0.2.12 tag with no changes of ours**, on a box with a Raphael iGPU (gfx1036) beside the engine's
Arc:

```
$ python -m unittest serve.test_server.GpuChoice.test_hip_ordinal      # tag 146df3c46, unmodified
FAILED (failures=1)          AssertionError: '0,1' != '1,0'
```

**After:** the class runs `Ran 11 tests ... OK`, and the full suite is `Ran 1535 tests ... OK (skipped=17)`.

## Approach

Twenty lines in `serve/test_server.py`, no engine or server change:

* the split case stubs `hip_gpu_archs` and `hip_speed_scores` — the two probes — so the assertion covers what it was
  written for: *a layer split keeps the config's list when the cards cannot be measured*;
* the deliberate reorder is then asserted **through the seam `ordered_gpus()` already exposes for tests**
  (`archs=`/`scores=`): `[1, 0]` with an integrated `gfx1036` becomes `[0, 1]`;
* the documented escape hatch is pinned in the same test: `"gpu_order": "as_given"` keeps `[1, 0]`.

So both behaviours are explicit, and neither depends on which cards the machine running the tests happens to have. No
production behaviour changed — if anything the reorder is now covered where it used to be only a comment.

## Validation

**Passed:**

* On a pristine `v0.2.12` worktree: `serve.test_server.GpuChoice` → `Ran 11 tests ... OK`.
* On the same worktree before the change: `test_hip_ordinal` → `FAILED (failures=1)` (the A/B above).
* Full suite on a source tree carrying this change: `Ran 1535 tests in 611.4 s ... OK (skipped=17)` — no failures, and
  the only difference from before is this test going green.

**Failed:** the unmodified tag, on this host only, for the reason described.

**Not run:**

* No Windows host was available, so the `hip_ordinal` path this test is named after (#325: the Windows display-adapter
  order, where an iGPU takes 0) is only exercised with stubbed scores on Linux. A Windows run would confirm the same
  numbers without the stubs.
* No machine with two discrete AMD cards was available, so the equal-cards branch of `ordered_gpus` was exercised only
  through injected scores.

## Limits

* This is a test-only change: it neither fixes nor changes any GPU selection behaviour. If the intended rule for a
  *manual* two-card split is "the config's order always wins", that is a different change — `"gpu_order": "as_given"`
  is today's way to say it, which is why the test now asserts it explicitly.
* The stubs hide the driver, which is the point; a regression in `hip_gpu_archs()` itself would no longer be caught by
  this test (it is caught by `pick_hip_devices`' own tests).

AI assistance was used to locate the cause and run the A/B; the failing and passing commands are the ones shown above,
run on this host.

Related on this site

Quick links to related pages.