#1261 461

Description

**Reopening: still reproducible on 0.1.40 with the bundled runtime provably in place (RX 7800 XT / gfx1101)**

Same signature as the original report, but now with the 0.1.35 fix present and demonstrably not
taking effect. You asked to be told when the GEMM error is still there with the bundled runtime —
here it is, with the `AMD_LOG_LEVEL=3` output.

**Environment**

- Windows 11 x64; AMD Ryzen 5 7600 (AVX-512); 31.1 GB RAM
- AMD Radeon RX 7800 XT, 16 GB, **gfx1101**, wave32, 30 multiprocessors (iGPU gfx1036 ignored)
- AMD display driver 32.0.31041.1004
- Strata **0.1.40**, ready-made AMD engine (`BUILD.json`: prebuilt, hip, windows-x64,
  rocm 10.2.0a20260930, hipblaslt_version 100500)
- Model: Coder IQ1_M, `--max-context 65536 --kv int8 --resident-experts --expert-cache auto
  --prefill auto --spec 4 --spec-min-p 0.5`
- Installed with `START-HERE.bat --yes --family coder --model IQ1_M --no-start` (completed cleanly)

**Symptom**

Short prompts answer correctly (best measured 43.9 tok/s). Any sizeable prefill kills the engine:

```
{"error": {"type": "server_error",
           "message": "the engine stopped unexpectedly (exit code 1); the next request restarts it"}}
```

Deterministic: a ~25-token prompt works, a ~1900-token prompt exits 1 every time. The bundled web
UI dies on its first request (long system prompt).

**The bundled runtime is present, correct, and still not the one loaded**

```
engine\amdhip64_7.dll          451d6b861610292e903b303bd0d9883a  16 399 360 B  (file ver 10.0.3686.0)
engine\rocm\bin\amdhip64_7.dll 451d6b861610292e903b303bd0d9883a  16 399 360 B   <- identical
engine\amd_comgr.dll           d43d0c1e6ad563d0e409ad4fbf34137d  124 121 600 B
engine\rocm\bin\amd_comgr.dll  d43d0c1e6ad563d0e409ad4fbf34137d  124 121 600 B   <- identical
C:\WINDOWS\SYSTEM32\amdhip64_7.dll  f5a36b72770ea6293d1a399ab7189ed9  (file ver 10.0.3679.0)
```

Both DLLs next to `strata.exe` are byte-identical to the `rocm\bin` originals, so the 0.1.35 fix is
installed. The engine still loads the driver's copy:

```
strata generate: GPU 0: AMD  Radeon RX 7800 XT (gfx1101)
strata generate: HIP runtime C:\WINDOWS\SYSTEM32\amdhip64_7.dll
...
strata generate: session is up (engine 0.1.40)
prefill gemm: dense MMQ on (STRATA_DENSE_MMQ=1)
rocBLAS error from hip error code: 'hipErrorInvalidDeviceFunction':98
prefill gemm: cublasGemmEx: cuBLAS status 6
```

`AMD_LOG_LEVEL=3`, from `engine\strata-device.exe --list-devices`:

```
:3:comgrctx.cpp            :126 : ...:  Loaded COMGR library version 3.0.
:3:hip_context.cpp          :66 : ...:  HIP Version: 7.2.60201.38d754472, Direct Dispatch: 0
:3:os_win32.cpp            :692 : ...:  HIP Library Path: C:\WINDOWS\SYSTEM32\amdhip64_7.dll
```

**What I tried, none of which changes the loaded runtime**

- Clean restart, all `python`/`strata` processes killed first.
- `lib_dirs: ["D:\\Strata\\engine\\rocm\\bin"]` (prepends the bundled dir to the engine's PATH).
- `D:\Strata\engine\rocm\bin` first on PATH manually before `strata-device.exe` — same line.
- `STRATA_DENSE_MMQ=1` — the switch takes effect (`prefill gemm: dense MMQ on` is printed) and the
  crash is unchanged, so it is not a workaround here.
- `HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs` has no AMD/HIP entry.
- `strata-device.exe --list-devices` and `--selftest` both pass (the selftest only exercises the
  VRAM arena) and report the same System32 runtime.

**Hypothesis**

The engine loads `dxgi.dll` (`src/platform/memory.cpp:70`). On an AMD machine that brings the
driver's user-mode graphics components into the process, and the driver package ships its own
`amdhip64_7.dll` in `System32`. Windows binds modules **by base name**: once that copy is loaded,
the engine's own import of `amdhip64_7.dll` resolves to it, and the copy next to `strata.exe` is
never considered — which would explain why the app-directory placement and every PATH change are
inert. If so, loading the bundled runtime explicitly by full path
(`LoadLibraryExA(..., LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR)`) before DXGI would fix it.

**Second, separate issue: the low-RAM resident mode never fits**

At engine start, available commit is ~3.0-3.5 GiB while the resident complement needs ~15.7 GiB:

```
FileExpertSource: room for the complement: 0.00 GiB (3.50 GiB available commit minus 4 GiB headroom
                  and a 0.25 GiB margin)
strata generate: WARNING: the resident RAM mode does not fit (...); the experts the GPU does not hold
                 are read from the model folder through the OS file cache (--mmap-experts)
```

The page file is 6.5 GB (system-managed) and the commit limit is 37.6 GB. The documented 32 GB /
`--family coder` tier ("the GPU holds ~45% of the experts, the other ~13 GB stay in RAM") never
happens: the log shows `files 102912.8 MB read` after only four short requests.

---

[strata-coder-iq1_m.log](https://github.com/user-attachments/files/33128675/strata-coder-iq1_m.log)