#461 Windows AMD (RX 7900 XTX / gfx1100): prefill GEMM crashes with hipErrorInvalidDeviceFunction

Beschreibung

## Summary

On Windows 11 with an AMD Radeon RX 7900 XTX (gfx1100), Strata 0.1.34 starts successfully, loads the IQ2_XS model, passes `strata-device --selftest`, and can generate responses for short prompts.

However, slightly larger prompts reproducibly crash the engine during prefill with:

```text
rocBLAS error from hip error code: 'hipErrorInvalidDeviceFunction':98
prefill gemm: cublasGemmEx: cuBLAS status 6
```

The API then returns HTTP 503:

```json
{
  "error": {
    "type": "server_error",
    "message": "the engine stopped unexpectedly (exit code 1); the next request restarts it"
  }
}
```

This does not appear to be specific to long-context / 32K prompts. I originally encountered it while testing a ~32K workload, but was able to reproduce the same crash with much shorter prompts.

## Environment

- OS: Windows 11 64-bit
- Strata commit: `1678de333d0e0711bc414ad992b640e1a37dd814`
- Strata engine: `0.1.34`
- Backend: HIP
- GPU: AMD Radeon RX 7900 XTX 24 GB
- Architecture: `gfx1100`, wave32
- CPU: AMD Ryzen 7 9800X3D
- RAM: 64 GB DDR5
- AMD display driver: `32.0.31041.1004`
- Driver date reported by Windows: `2026-08-17`
- ROCm libraries shipped with the Windows Strata engine: `10.2.0a20260930`
- Model: `Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS`
- Context: `32768`
- KV: 8-bit
- Vision: disabled

There is also an integrated AMD GPU in the system:

```text
device 0: AMD Radeon(TM) Graphics
  arch gfx1036, 36.2 GiB, wave32
  cannot run: GPU 0 (...) is not an architecture this Strata engine was compiled for

device 1: AMD Radeon RX 7900 XTX
  arch gfx1100, 24.0 GiB, wave32
```

Strata setup correctly selects the RX 7900 XTX as HIP device 1.

## Device self-test

I explicitly selected the RX 7900 XTX:

```powershell
$env:HIP_VISIBLE_DEVICES = "1"
.\engine\strata-device.exe --selftest
```

Result:

```text
device 0: AMD Radeon RX 7900 XTX
  HIP arch            gfx1100 wave32 (compiled for gfx1100,gfx1101,gfx1102,gfx1200,gfx1201,gfx1030)
  multiprocessors     48
  VRAM total / free   23.984 GiB (25753026560 B) / 23.839 GiB (25596395520 B)
  driver / runtime    70260201 / 70260201

memory plan  --max-context 20480
  KV + indexer keys   0.267387 GB
  recurrent state     0.000000 GB
  expert cache        5.675613 GB
  VRAM pool           5.943000 GB
  expert arena (DRAM) 0.000000 GB
  card free           23.839 GiB (25596395520 B)
  plan + KV vs free   FITS

selftest: arena 0.062 GiB (67108864 B), used 0.002 GiB (2097152 B) after two 1 MiB allocations
selftest: over-allocation refused as required
strata-device selftest OK
```

So basic HIP device detection and the Strata device self-test succeed.

## Successful short inference

A normal short request succeeds:

```text
strata serve: prompt 53 tokens = 0 reused + 53 read in 634 ms (83.6 tok/s), 47 generated in 775 ms (60.6 tok/s), drafts accepted 30 of 32, 1 checkpoints
strata serve: decode expert cache hit rate: 88.5% (19579 hits / 22118 lookups)
```

I also tested exact input-token counts using `/v1/messages/count_tokens`.

Independent short prompts succeeded at approximately 60-62 input tokens:

```text
60 input tokens: OK
61 input tokens: OK
62 input tokens: OK
```

## Reproduction

I generated independent prompts with different prefixes to avoid prefix-cache reuse.

Observed results:

```text
actual input tokens 61 -> OK
actual input tokens 62 -> OK
actual input tokens 61 -> OK
actual input tokens 72 -> HTTP 503 / engine exit
```

The 72-token request causes the engine to terminate.

Earlier tests with larger prompts, including a ~32K workload, fail with the same error.

This suggests the issue is not limited to a full 32K context. It can happen during a relatively short prefill as well.

I did not continue narrowing the exact token boundary because the crash was already reproducible enough to report.

## Engine log at failure

Immediately before the failure:

```text
strata generate: GPU 0: AMD Radeon RX 7900 XTX (gfx1100)
strata generate: loaded 33.02 GiB at 5.86 GiB/s
strata generate: expert cache auto: 19.19 GiB free, 700 MiB reserved (+143 MiB for the draft head) -> 13053 slots
strata generate: expert cache 13696 slots, 18.36 GiB of VRAM
strata generate: session is up (engine 0.1.34)
strata generate: token graph hit path: 13696 resident experts, decided on the device
strata serve: prompt chunk auto: 8192 tokens
strata serve: the prompt path borrows 3342 CUDA0 cache slots (4.49 GiB)
strata serve: 477 MiB of VRAM free with everything loaded

rocBLAS error from hip error code: 'hipErrorInvalidDeviceFunction':98
prefill gemm: cublasGemmEx: cuBLAS status 6
```

The same two error lines appear consistently after the failing requests.

I did not see an out-of-memory error in the engine log.

## Additional observation

The PCIe probe also reports values that appear implausibly high:

```text
PCIe probe: 9038.2 GB/s host->device
PCIe probe: 10294.7 GB/s host->device
PCIe probe: 19416.7 GB/s host->device
PCIe probe: 18417.5 GB/s host->device
PCIe probe: 5598.2 GB/s host->device
```

I do not know whether this is related to the prefill failure, but I included it in case it is useful for debugging.

## Related issues checked

I searched the existing issues before filing this.

The closest one I found is #273:

- RX 7900 XTX / gfx1100 / HIP
- Linux
- engine 0.1.29
- failure in the verify kernel (`unspecified launch failure`)

This report is different:

- Windows 11
- engine 0.1.34
- Windows ready-made HIP engine
- model loading and `strata-device --selftest` succeed
- short inference succeeds
- failure occurs in prefill GEMM
- error is `hipErrorInvalidDeviceFunction`

I also found #161, which has a somewhat similar high-level symptom where short prompts succeed and batched prefill fails, but that report is on NVIDIA/CUDA with a different cuBLAS status.

So I do not think this is a duplicate of either issue.

## Expected behavior

Prefill should complete successfully on the RX 7900 XTX / gfx1100 rather than terminating the engine.

## Actual behavior

The engine exits with:

```text
rocBLAS error from hip error code: 'hipErrorInvalidDeviceFunction':98
prefill gemm: cublasGemmEx: cuBLAS status 6
```

and the API responds with HTTP 503.

I can provide the full `strata-iq2_xs.log` or run additional targeted diagnostics if useful.