#5738 [Engine] Windows HIP (gfx1030): STRATA_ADAPT_OVERLAP=1 fails the first request with "never rang (unspecified launch failure)" when the whole expert arena is pinned

open · @ByteJuggler · 1 comments · View on GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

With `STRATA_ADAPT_OVERLAP=1` on Windows HIP, the first request after an engine start fails and the engine ends:

```
strata serve: fatal GPU error (verify: layer 11 never rang (unspecified launch failure)): the engine ends so that the server can start it again
```

The server starts it again. After 3 fatal errors in 10 minutes it stops. I expected the requests to run, as they do
without the setting. The layer number varies (7, 10 and 11 seen). Each crash also causes one Windows display-driver
timeout (System event 4101, amdkmdag) and the AMD bug-report popup.

With `STRATA_ADAPT_SIGNAL=copy` the text is the one from #2188:

```
strata serve: fatal GPU error (verify: the adaptive tier's copies for layer 11 never landed; its GPU waits were released and the GPU finished (#267) (the copy stream itself is not drained: reached 10 of 11, STRATA_VERIFY_TRACE=1 prints where the stream stopped)): ...
```

**How this differs from #2188.** That issue was a CUDA problem under WDDM: the stream-ordered signal's copies were
never submitted. 0.1.44 fixed it by making the host-function signal the Windows default and by flushing the copy
stream, and the reporter confirmed the fix on an RTX 4070 SUPER (23/23 turns clean with either signal). This report
is different in three ways:

1. It is HIP, not CUDA. The fix was confirmed on CUDA only.
2. The fix's own default, the host-function signal, is what fails here, and it fails with a launch failure. That
   looks like a fault on the GPU, not work that was never submitted.
3. It still happened after the fix: I reported "never rang" on 0.1.44.1 in #2188
   (https://github.com/Niko1221/Strata/issues/2188#issuecomment-6166484401). It was closed in the 0.2.0-beta reset
   without a test on AMD. It still happens on 0.2.11.

**It went with the pin of the expert arena.** These runs were a tuning sweep, one engine start per setting, with the
same requests each time (chat, agent and long prompts, about 30 minutes when nothing fails).

| Setting | Pin | Starts | Crashed |
|---|---|---|---|
| `STRATA_ADAPT_OVERLAP=1` | whole arena | 3 | 3 |
| `+ STRATA_ADAPT_SIGNAL=copy` | whole arena | 2 | 2 |
| `+ STRATA_COPY_HIP_PRIO=1` | whole arena | 3 | 3 |
| `+ STRATA_ADAPT_SIGNAL=copy` | partial (whole-arena pin refused, see below) | 1 | 0 |
| `STRATA_ADAPT_OVERLAP=1`, `STRATA_ARENA_PIN_GIB=32` | partial (forced) | 1 | 0 |
| `STRATA_ADAPT_OVERLAP=1`, `STRATA_ARENA_PIN_GIB=8` | partial (forced) | 1 | 0 |
| `STRATA_ADAPT_OVERLAP=1`, `STRATA_HIP_ADAPT_KERNEL_COPY=1` | whole arena | 1 | 1 |
| `STRATA_ADAPT_OVERLAP=1`, `STRATA_HIP_ADAPT_KERNEL_COPY=1` | partial (whole-arena pin refused) | 1 | 0 |
| `STRATA_ADAPT_OVERLAP=1`, `STRATA_HIP_ADAPT_KERNEL_COPY=1`, `STRATA_ADAPT_SIGNAL=copy` | whole arena | 3 | 0 |

- "Whole arena" means the engine log says `expert arena: cudaHostRegister PORTABLE ok`.
- The two unforced partial pins happened because the #3431 check refused the whole-arena pin (`alias [...] of the
  range at ... covers memory in use`), so the engine pinned slices instead.
- In every start, crashed or clean, the engine logged `adaptive overlap (STRATA_ADAPT_OVERLAP=0 turns it off): the
  adaptive tier's copies land while the next window runs`, so the overlap was on. Each clean start ran the whole set
  with no driver timeout.

**One combination works.** With both `STRATA_HIP_ADAPT_KERNEL_COPY=1` and `STRATA_ADAPT_SIGNAL=copy`, 3 of 3 whole-pin starts ran the whole set of tests clean. Either one alone still crashed attempting a whole pin. This combination brings no speed however: against the same config without adapt-overlap, all three runs were within 0-3% (chat 33.5-34.8 against 34.5 tok/s, agent 39.5-40.0 against 40.5, 16K prompt 970-973 against 976).

**CUDA does not crash.** On an RTX 3090 (Windows, CUDA, engine 0.1.57, IQ3_XXS) with the whole arena pinned and the
log confirming the overlap was on, 2 of 2 starts ran the same set clean, with the default signal and with
`STRATA_ADAPT_SIGNAL=copy`. No speed gain there either.

**A partial pin is not a useful workaround.** With the pin capped at 32 GiB, adapt-overlap was no faster than the same
partial pin without it. Both are much slower (on this machine at least) on prompts than a whole pin:

| Run | Chat decode | Agent decode | 4K prompt | 16K prompt |
|---|---|---|---|---|
| adapt-overlap, pin capped at 32 GiB | 31.1 | 37.1 | 442 | 672 |
| no adapt-overlap, partial pin (31 GiB) | 30.6 | 36.9 | 441 | 666 |
| no adapt-overlap, whole pin (2 runs) | 34.4 | 41.1 | 587 | 893 |

(tok/s, medians over the set. The pin capped at 8 GiB was slower again: chat 26.2.)

**What this might mean (unconfirmed, from these runs only).** On Windows HIP, the overlapped SDMA copies of the
adaptive tier fail when their source sits in one large registered range, and work when the arena is registered in
slices. On HIP every registered range gets a second GPU address (#3431), and CUDA has no such alias, which fits CUDA
running clean on a whole pin. Copying with a kernel takes SDMA out, but that alone was not enough: the host-function
signal still failed until the copy signal replaced it.

Possibly related, all AMD and all in or near the adaptive tier's copies:
- #5672: Windows HIP, gfx1100. `--adapt-async 1` gives a verify timeout plus event 4101 during long answers.
- #5358: gfx1030 on Linux. Fatal GPU faults on host addresses with adaptive swaps, even with
  `STRATA_HIP_ADAPT_KERNEL_COPY=1`.
- #4591: R9700 on Linux. SDMA copies fault on host addresses.

Windows HIP doesn't print a faulting address, so I can't tell whether this is the same fault.

### Smallest steps or command that reproduce it

1. Windows HIP engine on an RX 6900 XT, IQ3_S, with the expert arena pinned whole. Check the engine log for
   `expert arena: cudaHostRegister PORTABLE ok`.
2. Add `"STRATA_ADAPT_OVERLAP": "1"` to the config's `env` and start the server.
3. Send any chat request. It fails with the error above. In 9 of 9 whole-pin crashes it was the first request.

To see it run clean, add `"STRATA_ARENA_PIN_GIB": "32"` to the same config, or keep the whole pin and add both
`"STRATA_HIP_ADAPT_KERNEL_COPY": "1"` and `"STRATA_ADAPT_SIGNAL": "copy"`.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.11 · engine 0.1.57, the release package `strata-engine-0.1.57-windows-x86_64-hip-rdna2.tar.gz` (commit 0bdcef4682c5, ROCm 10.2.0a20260930)

### Graphics card, memory and system

RX 6900 XT 16 GB (gfx1030 / RDNA2) · 192 GB RAM (DDR4-2400) · i7-5930K · Windows 11 Pro 23H2

### Model, size and settings

Qwen3.8-Flash-Next IQ3_S (GSQ-RCO pack) · one GPU · expert arena about 47 GiB in RAM · large pages not available (refused by Windows) · `STRATA_HIP_PROMPT_F16=1`

### Logs

The fatal-error lines are quoted above. Full engine and server logs for each start are available if they help.

Related on this site

Quick links to related pages.