#2781 [BUG] Engine crashes with 0xC0000005 in hipModuleLoad on first request — AMD RX 9070 XT (gfx1201) on Windows, Strata 0.1.44.1

Description

## Summary

On Windows + AMD Radeon RX 9070 XT (gfx1201), the engine starts cleanly to `session is up` but crashes on the first user request with an access violation (0xC0000005) inside `amdhip64_7.dll` → `libtensilelite-host.dll` → `libhipblaslt.dll hipblasLtMatmul` → `rocblas_gemm_ex`. The crash is at the `hipModuleLoad` boundary on every code-object load — not specific to one kernel. The bundled ROCm (`engine/amdhip64_7.dll`, `engine/rocm/bin/*`) appears to have a Windows + gfx1201 incompatibility in the hipBLASLt / TensileLite path.

## Environment

- **Model**: `qwen3.8-flash-next-iq2_xs` (the recommended `--family qwen --model IQ2_XS` on 48 GB RAM)
- **Strata version**: `0.1.44.1` (latest on `main` at the time of this report; `MIN_ENGINE = (0, 1, 44, 1)`)
- **GPU**: AMD Radeon RX 9070 XT, 16 GB GDDR6, gfx1201 (RDNA4)
- **Driver**: `32.0.31036.15` — AMD Software: PRO Edition 26.Q3 (released 2026-08-12). This is the latest PRO Edition for this card; the consumer Adrenalin 26.9.2 (Optional) is also available but I have not installed it.
- **CPU**: AMD Ryzen 7 9700X (8C/16T, AVX-512)
- **RAM**: 48 GB DDR5 (system shows 47 GB usable)
- **OS**: Windows 10 64-bit (the "Windows 10 - 64-Bit Edition" download section on amd.com)
- **Other GPU on the box**: integrated `AMD Radeon(TM) Graphics` (gfx1036, the Ryzen APU) — not used by Strata (it's not in the engine's gfx list); `engine/strata-device.exe --list-devices` correctly picks GPU 0 (9070 XT).

## Repro steps

1. Fresh install with `START-HERE.bat --yes --family qwen --model IQ2_XS --no-start`.
2. Run `run-iq2_xs.bat`. Engine starts, reaches `session is up (engine 0.1.44.1)`.
3. Open the web UI (`http://127.0.0.1:8080`) and send any message (e.g. "hello").
4. Engine dies mid-first-request. Web UI shows `the engine stopped unexpectedly (exit code 3221225477)` (= 0xC0000005 in signed form).
5. Server auto-restarts the engine. The same prompt hits the same crash on every restart.

## Crash stack (verbatim, from `strata-iq2_xs.log`)

The crash consistently appears at the `hipModuleLoad → TensileLite → hipBLASLt → rocBLAS` boundary. Last full stack:

```
Exception Code: 0xC0000005
0x00007FFEF984A7FE, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x83A7FE, hsa_system_get_major_extension_table() + 0x3AD1DE
0x00007FFEF9455785, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x445785, hipProfilerRegisterChunkCallbackExt() + 0x9EF05
0x00007FFEF94702D0, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x4602D0, hipProfilerRegisterChunkCallbackExt() + 0xB9A50
0x00007FFEF946FA39, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x45FA39, hipProfilerRegisterChunkCallbackExt() + 0xB91B9
0x00007FFEF97ECB8D, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x7DCB8D, hsa_system_get_major_extension_table() + 0x34F56D
0x00007FFEF97EACB5, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x7DACB5, hsa_system_get_major_extension_table() + 0x34D695
0x00007FFEF97E99A5, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x7D99A5, hsa_system_get_major_extension_table() + 0x34C385
0x00007FFEF947115E, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x46115E, hipProfilerRegisterChunkCallbackExt() + 0xBA8DE
0x00007FFEF93D85F9, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x3C85F9, hipProfilerRegisterChunkCallbackExt() + 0x21D79
0x00007FFEF93D07C7, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x3C07C7, hipProfilerRegisterChunkCallbackExt() + 0x19F47
0x00007FFEF93DD801, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x3CD801, hipProfilerRegisterChunkCallbackExt() + 0x26F81
0x00007FFEF908B0A4, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x7B0A4, amd_dbgapi_get_build_id() + 0x4E264
0x00007FFEF901B341, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0xB341, hipGetCmdName() + 0x7AB1
0x00007FFEF92BE973, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x2AE973, hipRegisterTracerCallback() + 0x1371A3
0x00007FFEF92A5D9D, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x295D9D, hipRegisterTracerCallback() + 0x11E5CD
0x00007FFEF937FC73, .../engine/amdhip64_7.dll(0x00007FFEF9010000) + 0x36FC73, hipModuleLoad() + 0x33
0x00007FFEF8B88967, .../engine/rocm/bin/libtensilelite-host.dll(0x00007FFEF8970000) + 0x218967, SolutionAdapter::loadCodeObjectFile() + 0x67
0x00007FFEF8B8BD39, .../engine/rocm/bin/libtensilelite-host.dll(0x00007FFEF8970000) + 0x21BD39, SolutionAdapter::FindCodeObject() + 0x6B9
0x00007FFEF8B8CEBC, .../engine/rocm/bin/libtensilelite-host.dll(0x00007FFEF8970000) + 0x21CEBC, SolutionAdapter::launchKernel() + 0x3C
0x00007FFEF8B8DBC0, .../engine/rocm/bin/libtensilelite-host.dll(0x00007FFEF8970000) + 0x21DBC0, SolutionAdapter::launchKernels() + 0x60
0x00007FFEF8DBE041, .../engine/rocm/bin/libhipblaslt.dll(0x00007FFEF8D90000) + 0x2E041
0x00007FFEF8DAD107, .../engine/rocm/bin/libhipblaslt.dll(0x00007FFEF8D90000) + 0x1D107
0x00007FFEF8DB157E, .../engine/rocm/bin/libhipblaslt.dll(0x00007FFEF8D90000) + 0x2157E
0x00007FFEF8F28E79, .../engine/rocm/bin/libhipblaslt.dll(0x00007FFEF8D90000) + 0x198E79, hipblasLtMatmul() + 0xF9
0x00007FFE46D7EEBB, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x5FEEBB
0x00007FFE46DD1DE4, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x651DE4
0x00007FFE46F30BDA, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x7B0BDA, rocblas_internal_gemm_grouped_batched_ex_template<H>() + 0x1297A
0x00007FFE46F307C4, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x7B07C4, ... + 0x12564
0x00007FFE46F262EA, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x7A62EA, ... + 0x808A
0x00007FFE46F23F13, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x7A3F13, ... + 0x5CB3
0x00007FFE46F00DF6, .../engine/rocm/bin/rocblas.dll(0x00007FFE46780000) + 0x780DF6, rocblas_gemm_ex() + 0x366
0x00007FFF04FA0F45, .../engine/rocm/bin/hipblas.dll(0x00007FFF04F50000) + 0x50F45, hipblasGemmEx() + 0x2F5
0x00007FF7BC252485, .../engine/strata.exe(0x00007FF7BBF00000) + 0x352485
0x00007FF7BC0D1A74, .../engine/strata.exe(0x00007FF7BBF00000) + 0x1D1A74
0x00007FF7BC0BB89C, .../engine/strata.exe(0x00007FF7BBF00000) + 0x1BB89C
0x00007FF7BC0D51E8, .../engine/strata.exe(0x00007FF7BBF00000) + 0x1D51E8
0x00007FF7BBF7088C, .../engine/strata.exe(0x00007FF7BBF00000) + 0x7088C
0x00007FF7BBF2AB07, .../engine/strata.exe(0x00007FF7BBF00000) + 0x2AB07
0x00007FF7BC679C6C, .../engine/strata.exe(0x00007FF7BBF00000) + 0x779C6C
0x00007FFF38517374, KERNEL32.DLL, BaseThreadInitThunk() + 0x14
0x00007FFF38FBCC91, ntdll.dll, RtlUserThreadStart() + 0x21
```

The same stack appeared earlier as `hipModuleUnload failed: unspecified launch failure` + `Exception Code: 0xC00000FD` (stack overflow) — that's the same path before `ROCBLAS_WORKSPACE_STACK_SIZE` was raised; the underlying fault is in `hipModuleLoad`, not in stack exhaustion. Once the rocBLAS worker threads had enough stack, the crash turned into the clean 0xC0000005 in `hipModuleLoad` itself.

## Engine output before crash

```
strata generate: GPU 0: AMD Radeon RX 9070 XT (gfx1201)
strata generate: HIP runtime E:\AI\Strata\engine\amdhip64_7.dll
strata generate: expert arena: locked 10901 MiB via working-set minimum + VirtualLock; cudaHostRegister of the whole arena FAILED (invalid argument); 33 slices pinned (22 GiB); large pages refused for 35456548864 B (GetLargePageMinimum=2097152, VirtualAlloc error 1450); using 4 KB pages
strata generate: expert cache 6472 slots, 8.67 GiB of VRAM
strata serve: 2289 MiB of VRAM free with everything loaded
... crash before any "prompt X tokens" line ...
```

The engine always reaches "VRAM free with everything loaded" then dies before the first prefill. Speed numbers are reachable only via the first request of an engine instance — i.e. never, in practice.

## Things I tried (none helped)

1. Removed the `--control-vector-scaled ... --cvec-mode project` lines from `strata-iq2_xs.json` (no longer needs the `data/Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf` steering vector). Same crash, different stack frame ordering but same `hipModuleLoad` top.
2. Reduced `--max-context` from 131072 to 32768. Same crash.
3. Set `ROCBLAS_WORKSPACE_STACK_SIZE=33554432` and `HIPBLASLT_WORKSPACE_STACK_SIZE=33554432` in the config's `"env"` field (passed through `serve/server.py:child_env` → `strata.exe`). The earlier `Exception Code: 0xC00000FD` (STATUS_STACK_OVERFLOW) disappeared and became the `0xC0000005` you see above. So the rocBLAS stack fix does work; the remaining crash is inside `hipModuleLoad` itself.
4. Set `STRATA_GFX12_DEFAULTS=0` in `"env"` — confirmed by `arch_defaults.cpp` reading `getenv("STRATA_GFX12_DEFAULTS")` (engine no longer prints the `"5 exact speed switches on by default"` line). The five `STRATA_GDN_*` / `STRATA_CVEC_FUSE` defaults are now off. No effect on the crash.
5. Set `STRATA_WDDM_BUDGET=0`. No effect on the crash (used ~11 GiB of expert cache regardless).
6. Verified `engine/strata-device.exe --list-devices` sees the card correctly:

   ```
   device 0: AMD Radeon RX 9070 XT
     arch gfx1201, 15.9 GiB, wave32
   ```

   (device 1 is the integrated gfx1036 APU which the bundled ROCm was not built for, as expected — Strata picks device 0 only.)

## Side observations

- `expert arena: large pages refused for 35456548864 B (GetLargePageMinimum=2097152, VirtualAlloc error 1450); using 4 KB pages` — the bundled ROCm explicitly falls back to 4 KB pages on Windows. This may contribute to the code-object load path being exercised on every request (no in-RAM code cache shared across invocations?).
- The 0xC00000FD → 0xC0000005 transition after raising the rocBLAS worker-stack size strongly suggests the bundled `engine/rocm/bin/*` does deep template instantiation in `rocblas_internal_gemm_grouped_batched_ex_template<H>` which on Windows + gfx1201 is right at the limit of the 8 MiB default thread stack. Even with `ROCBLAS_WORKSPACE_STACK_SIZE` raised, the launch eventually falls over inside `hipModuleLoad`.
- On a previous session before I disabled `STRATA_GFX12_DEFAULTS=0`, the crash surfaced as `prefill mmq: iota: unspecified launch failure` (a different failure point but in the same launch family) — toggling the gfx12 defaults moves the failure around but does not eliminate it.

## Suggestion

The bundled ROCm (`engine/rocm/bin/*`, `engine/amdhip64_7.dll`, `libtensilelite-host.dll`, `libhipblaslt.dll`, `rocblas.dll`, `hipblas.dll`) on Windows appears not to be combined with a shipped code-object cache (the equivalent of `~/.cache/hipblaslt` on Linux). On Windows + gfx1201 this means every first request of every engine instance triggers a fresh `hipModuleLoad` → TensileLite JIT path. If there is an opt-in env var to disable hipBLASLt and force a non-JIT path on Windows, or to keep the code-object cache warm across requests, that may be the fix.

## Files / logs

- `E:\AI\Strata\strata-iq2_xs.log` — approximately 1840 lines, multiple crashes; the last crash block ends at the line before the next `[strata] ... engine started:`.
- `E:\AI\Strata\strata-iq2_xs.json` and `E:\AI\Strata\run-iq2_xs.bat` — the config I ran.

[strata-iq2_xs.log](https://github.com/user-attachments/files/33553651/strata-iq2_xs.log)
[strata-iq2_xs.json](https://github.com/user-attachments/files/33553671/strata-iq2_xs.json)

## Environment summary one-liner

`Win10 64-bit · AMD Radeon RX 9070 XT · 32.0.31036.15 (PRO 26.Q3) · Strata 0.1.44.1 · Qwen3.8-Flash-Next IQ2_XS · crash in hipModuleLoad on first request`