Issues / #1785
#1785 [Bug] Windows HIP (Radeon AI PRO R9700, gfx1201): engine hangs at verify-window capture at `--max-context 262144` — LiveKernelEvent 141 (TDR); 229376 and below start and serve fine
open · @Metalazzo · 0 comentarios · En GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descripción
## Summary
On Windows 10 with the prebuilt HIP engine, **every start at `--max-context 262144` dies during the verify-window capture phase** at serve init: the GPU wedges, Windows logs `LiveKernelEvent 141` (watchdog reset) twice, and the engine exits. Four consecutive starts at 262144 all failed the same way (across two engine versions). The same config at `--max-context 229376` starts normally, captures all six windows, and served a real 119,136-token prompt without issue. Free VRAM at the failure point is identical between the two settings (2308 vs 2306 MiB), so this is not VRAM pressure — something about 262144 specifically wedges the driver during capture.
## Environment
- **OS:** Windows 10 Professional 22H2 (10.0.19045)
- **Engine:** ready-made `strata-windows-x64-hip.zip` — `BUILD.json`: prebuilt, hip, windows-x64, **0.1.41**, rocm 10.2.0a20260930, hipblaslt_version 100500, archs gfx1100/1101/1102/1200/1201/1030/1151
- **GPU:** Radeon AI PRO R9700 32 GB (gfx1201); a second card (RX 9070 XT, gfx1201) present and idle; 9070XT card drive 2 monitors (2560 px / 1920 px)
- **AMD driver:** 32.0.23033.1002, dated 2026-03-09 — deliberately kept: newer Adrenalin builds are reported (community) to have a bug that empties/resets VRAM while it is used by an LLM workload, so I have not updated. The driver predates the bundled ROCm 10.2.0a20260930; if a specific driver version is known-good with this ROCm, please name it.
- **RAM:** 96 GB RAM, page file 96 GB (commit limit ≈ 192 GB, ~68 GB free at crash time)
- **TDR:** `TdrDelay=20`, `TdrDdiDelay=10` already set in `GraphicsDrivers` (the hang exceeds 20 s)
- **Model:** Qwen3.8-Flash-Next GSQ-RCO **IQ3_S** native pack, MTP rt, expert-profile
- flags: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 32768`, hipBLASLt table `gfx1201-hipblaslt-100500.txt`
## Steps to reproduce
The server restarts the engine each time (see timeline). Exact json:
```bash
{
"exe": "J:\\Strata-main\\engine\\strata.exe",
"args": [
"--pack",
"J:\\Strata-data\\packs\\iq3_s",
"--native",
"J:\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
"--ple-gguf",
"J:\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
"--expert-profile",
"J:\\Strata-main\\data\\expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--spec-min-p",
"0.5",
"--mtp",
"J:\\Strata-data\\mtp\\rt",
"--max-context",
"229376",
"--kv",
"int8",
"--kv-resident",
"32768"
],
"cwd": "J:\\Strata-main",
"tokenizer": "J:\\Strata-data\\packs\\iq3_s\\tokenizer",
"model_name": "qwen3.8-flash-next-iq3_s",
"log": "J:\\Strata-main\\strata-iq3_s.log",
"lib_dirs": [
"J:\\Strata-main\\engine\\rocm\\bin"
],
"port": 8080,
"host": "0.0.0.0",
"backend": "hip",
"env": {
"STRATA_HIPBLASLT_TUNING": "J:\\Strata-main\\tools\\hip\\gfx1201-hipblaslt-100500.txt"
},
"gpu": 1,
"gpus_asked": true
}
```
## Symptom (engine log, 0.1.41, 262144)
The line `strata verify: capturing the N-token window` never prints — the engine dies before/at the first capture.
```text
strata generate: expert cache auto: 25.10 GiB free, 2560 MiB reserved (+218 MiB for the draft head) -> 9029 slots
strata generate: session is up (engine 0.1.41)
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2302 CUDA0 cache slots (4.38 GiB)
pref gemm: hipBLASLt tuning enabled (32 rows, gfx1201, version 100500)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 2308 MiB of VRAM free with everything loaded
hipModuleUnload failed:
error: unspecified launch failure
Exception Code: 0xC00000FD
```
Windows Error Reporting for the same minute:
- `LiveKernelEvent` **141** at 20:37:10 and 20:37:38 (dump `WATCHDOG-20261009-2037.dmp`)
- `APPCRASH`: `strata.exe`, faulting module `Dbghelp.dll` 10.0.19041.7725, exception `0xc0000005` (the engine's own crash handler faulting; the `0xC00000FD` in the log looks secondary)
## Timeline (2026-10-09, server auto-restarts the engine each time)
| Start | Engine | max-context | Outcome |
|---|---|---|---|
| 20:26:08 | 0.1.40.2 | 262144 | died at serve init; log printed its own warning: `447 MiB of VRAM free … that is little room for the verify windows' buffers under WDDM` (the #1275 shape, 700 MiB default reserve) |
| 20:32:43 | 0.1.41 | 262144 | 2308 MiB free → died at first capture, `unspecified launch failure` + `0xC00000FD` |
| 20:37:10 | 0.1.41 | 262144 | same; LiveKernelEvent 141 ×2 |
| 20:38:07 | 0.1.41 | 262144 | same (died within ~50 s) |
| 20:39:00 | 0.1.41 | 196608 | fine, served for hours |
| 21:05:58 | 0.1.41 | **229376** | **fine**: all six windows captured (`upload no error, sync no error`), then a real prompt of 119,136 fresh tokens read at 305 tok/s, a second 121,020-token prompt with 119,403 reused, decode 52 tok/s, expert-cache hit 92.6 %, KV streaming 96 % VRAM hit |
## What rules out the obvious suspects
- **Not VRAM pressure:** 2308 MiB free at the 262144 crash vs 2306 MiB free at the working 229376 start — identical headroom; the 0.1.41 2560 MiB auto-reserve (#1376) is active in both.
- **Not the commit limit (#1607/#1630):** ~68 GB of commit free at the crash time; the crash is at serve init, before any prompt.
- **Not TDR being too short:** `TdrDelay` is already 20 s; the GPU is wedged for more than that.
- **Not model/context infeasible on this card:** the docs' own R9700 runs IQ3_S at 431072 ctx on Linux, and #1565 ran IQ3_S @ 262144 on gfx1201 (ROCm). It's the Windows/WDDM capture path.
## Not yet tried (happy to run anything)
`--prefill auto:4096` at 262144 (expect little given the identical VRAM), `--vram-reserve-mib 3584`, `STRATA_HIP_WMMA=1`, a newer AMD driver — held back for the VRAM-clearing bug described above.
## Related
#961 (WDDM verify window wedges the driver → 0x141 → TDR; there on NVIDIA), #1275 (the VRAM-tight variant, matches my first 0.1.40.2 start), #1630, #1644 (GPU faults on native IQ packs on Windows HIP).
## Question
If a specific AMD driver version is known-good with ROCm 10.2.0a20260930, please name it.
En el sitio
Enlaces a install, modelos, releases.