Issues / #3555

#3555 [Engine] Windows + RX 7900 XTX (gfx1100): engines 0.1.45.3 / 0.1.46.1 never become ready - "verify: stream create failed", HIP 719 in hipBLASLt init, ~1.9 GiB VRAM free, 0.1.43.1 works

open · @yifengyous · 2 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

### Desktop app, Terminal or source?

Source - a checkout of the tree, launched as `python serve/server.py --engine strata --config strata-unsloth-ud-iq4_xs.json --port 8084`. Not a git clone, no app and no Terminal; the run config is hand-written. Updating means replacing `engine/` and the `serve/` tree by hand (which is how I tested 0.1.45.3 and 0.1.46.1).

### What happened

0.1.43.1 works. Upgrading to 0.2.1 (**engine 0.1.46.1**, package `strata-engine-0.1.46.1-windows-x86_64-hip-rdna3`) the engine never becomes ready. It builds the whole startup - arena, session, native head, drafter, expert cache, prompt path - and then dies in the verify step:

```
strata trace:  1950 MiB free after the head and the prompt path
strata serve: verify: stream create failed
Hip error: 'unspecified launch failure'(719) at C:/home/runner/_work/rockrel/rockrel/rocm-libraries/projects/hipblaslt/library/src/amd_detail/hipblaslt.cpp:468
```

The same happens with **engine 0.1.45.3** (`strata-engine-0.1.45.3-windows-x86_64-hip-rdna3` from 0.2.0-beta), so the regression is not 0.1.46-only. With 0.1.45.3 the line before the failure differs, the failure does not:

```
prefill gemm: cannot open tuning file; using hipBLASEx
strata trace:  1968 MiB free after the head and the prompt path
strata serve: verify: stream create failed
Hip error: 'unspecified launch failure'(719) at .../hipblaslt.cpp:468
```

### Smallest steps or command that reproduce it

1. Windows source checkout of 0.1.43.1 works with this config: `--max-context 131072 --kv int8 --spec 4 --spec-min-p 0.7 --resident-budget-gib 44 --vram-reserve-mib 1856 --prefill auto --expert-cache auto`.
2. Replace `engine/` with `strata-engine-0.1.46.1-windows-x86_64-hip-rdna3` (or `0.1.45.3`), keep the same config and model.
3. Start via `serve/server.py`.
4. It loads everything (44 GiB pinned complement, expert cache 5702 slots / 12.85 GiB) and then exits with the two lines above.

Nothing else on the GPU; a fresh start each time; 10 attempts, all identical.

### Reproduction status

Reproduced locally - 10 fresh engine starts across the two engine versions (rows below).

### Strata version

Strata 0.2.1 - engine **0.1.46.1** (BUILD.json: hip, `hipblaslt_version 100500`, archs include gfx1100, rocm 10.2.0a20260930).
Also tested: engine **0.1.45.3** from 0.2.0-beta - fails identically.
**Working**: engine **0.1.43.1** with the same model, pack and config.

### Graphics card, memory and system

Radeon RX 7900 XTX 24 GB (**gfx1100**) - driver **32.0.31041.1004** (2026-08-17) - 64 GB DDR5-5600 (CL38) - Windows 11 Pro 25H2 build **26200.9711** - Ryzen 7 9700X - SAPPHIRE PULSE B850MP WIFI.

### Model, size and settings

Unsloth **UD-IQ4_XS** of Qwen3.8-Flash-Next, `--native` GGUF shards + pack (`--pack`), `--expert-profile`, `--mtp`, 131072 context, pictures off, `--resident-budget-gib 44`. Serving is fully resident on 0.1.43.1: `resident 42.6 GiB of experts in RAM` and **0 blob reads** from the file tier.

### Logs

Startup trace with `STRATA_TRACE=1` (engine 0.1.46.1, production config):

```
strata trace: 22169 MiB free after the CUDA context and the weight arena
strata trace: 22169 MiB free after the pack weights
strata trace: 17201 MiB free after the session (KV, GDN state, PLE history)
strata trace: 15633 MiB free after the native head
strata trace: 15620 MiB free after the weights, the session and the drafter
strata trace:  2003 MiB free after opening the expert cache
strata trace:  2010 MiB free after the profile fill
strata trace:  2010 MiB free after the R4 hit path
strata trace:  2010 MiB free after the expert cache and the graphs
strata trace:  2011 MiB free after before the prompt path init
prefill gemm: hipBLASLt tuning enabled (32 rows, gfx1100, version 100500)
strata trace:  1950 MiB free after the head and the prompt path
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
strata serve: verify: stream create failed
Hip error: 'unspecified launch failure'(719) at .../hipblaslt.cpp:468
```

With `STRATA_HIPBLASLT_VERBOSE=1`, right before the failure:

```
prefill gemm: hipBLASLt tuning enabled (32 rows, gfx1100, version 100500)
strata serve: verify: stream create failed
prefill gemm: hipBLASLt summary launches=0 fallbacks=0 unique_fallback_shapes=0
Hip error: 'unspecified launch failure'(719) at .../hipblaslt.cpp:468
```

`launches=0 fallbacks=0` - no GEMM was ever launched, so this is not a kernel for the wrong arch, it is the handle/stream creation.

### It is not memory

* `STRATA_TRACE=1` reports **1950-1968 MiB free** at the moment of the failure; the expert cache is sized normally (5702 slots, 12.85 GiB).
* Shrinking the prompt path changes nothing: with `--vram-reserve-mib 4096 --prefill 1024` the cache drops to 5175 slots / 10.63 GiB and the prompt path borrows only 0.86 GiB - same failure, ~1.97 GiB free.
* `HIP_LAUNCH_BLOCKING=1` does not move the failure earlier, so it is not a sticky error left by a previous asynchronous kernel.

### What I tried (each row a fresh engine start, everything else unchanged)

| # | Engine | Change | Result |
|---|---|---|---|
| 1 | 0.1.43.1 | production config | **works** (42.6 GiB resident, 0 blob reads, 50-67 tok/s) |
| 2 | 0.1.46.1 | production config | fail, same call |
| 3 | 0.1.46.1 | `--vram-reserve-mib 3072 --prefill 2048` | fail |
| 4 | 0.1.46.1 | `--vram-reserve-mib 4096 --prefill 1024` (cache 5175 slots / 10.63 GiB, borrow 0.86 GiB) | fail |
| 5 | 0.1.46.1 | `STRATA_PF_GEMM=0` | fail |
| 6 | 0.1.46.1 | `STRATA_HIP_QUEUE_PRIME=0` + `STRATA_PREFILL_STREAM_AHEAD=0` | fail |
| 7 | 0.1.46.1 | `STRATA_ARENA_PIN_GIB=16` (the fix that worked in #2013) | fail |
| 8 | 0.1.46.1 | `STRATA_HIPBLASLT_TUNING` unset | fail |
| 9 | 0.1.45.3 | production config | fail, same call |
| 10 | 0.1.45.3 / 0.1.46.1 | `HIP_LAUNCH_BLOCKING=1` + `STRATA_TRACE=1` | fail, error does not move |

### Why I think it is the prefill GEMM path

* 0.1.43.1 never prints a `prefill gemm:` line with this configuration and starts fine; 0.1.45.3 and 0.1.46.1 print it and die in `hipblaslt.cpp:468` with `launches=0`.
* The outcome is identical with the tuning table present, absent, or the package's own `engine/tuning/gfx1100-hipblaslt-100500.txt`.
* gfx1100 is supported by the shipped library: both engine packages carry `rocm/bin/hipblaslt/library/gfx1100` (98 files, incl. `Kernels.so-000-gfx1100.hsaco` and `hipblasltTransform_gfx1100.hsaco`), a gfx1100 tuning table, and the same `libhipblaslt.dll 1.5.0.0` as 0.1.43.1.

### Possibly related issues

* **#2013** - Windows Strix Halo (gfx1151), same `unspecified launch failure` class during serve init; `STRATA_ARENA_PIN_GIB=16` fixed it there. No effect here (row 7).
* **#3263** - 0.2.0-beta engine 0.1.45.3 on a real RX 7900 XTX (gfx1100) **on Linux** runs, same speed as 0.1.44.1. So gfx1100 itself is fine; the Windows build is where it breaks.
* **#2960**, **#3463** - other Windows + gfx1100 reports.
* 0.1.45 release notes: "Not run: Windows hardware (...), Intel Arc, the RTX 5070."

### Questions

1. Is there an environment switch that turns the hipBLASEx / hipBLASLt prefill path **completely** off? `STRATA_PF_GEMM=0` did not change the failure.
2. Is there a diagnostic build, or `STRATA_TRACE=2`, that prints which hipBLASLt call returns 719 (handle, stream, workspace query)?
3. Does 0.1.46.2+ touch this initialisation on Windows? I can test any engine package you point me at - the machine is free in the evenings and I keep a full rollback of 0.1.43.1.

Happy to run any diagnostic you want on this exact machine.

En el sitio

Enlaces a install, modelos, releases.