Issues / #1785

#1785 [Bug] Windows HIP (Radeon AI PRO R9700, gfx1201): engine hangs at verify-window capture at `--max-context 262144` — LiveKernelEvent 141 (TDR); 229376 and below start and serve fine

open · @Metalazzo · 0 Kommentare · Auf GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Beschreibung

## Summary

On Windows 10 with the prebuilt HIP engine, **every start at `--max-context 262144` dies during the verify-window capture phase** at serve init: the GPU wedges, Windows logs `LiveKernelEvent 141` (watchdog reset) twice, and the engine exits. Four consecutive starts at 262144 all failed the same way (across two engine versions). The same config at `--max-context 229376` starts normally, captures all six windows, and served a real 119,136-token prompt without issue. Free VRAM at the failure point is identical between the two settings (2308 vs 2306 MiB), so this is not VRAM pressure — something about 262144 specifically wedges the driver during capture.

## Environment

- **OS:** Windows 10 Professional 22H2 (10.0.19045)
- **Engine:** ready-made `strata-windows-x64-hip.zip` — `BUILD.json`: prebuilt, hip, windows-x64, **0.1.41**, rocm 10.2.0a20260930, hipblaslt_version 100500, archs gfx1100/1101/1102/1200/1201/1030/1151
- **GPU:** Radeon AI PRO R9700 32 GB (gfx1201); a second card (RX 9070 XT, gfx1201) present and idle; 9070XT card drive 2 monitors (2560 px / 1920 px)
- **AMD driver:** 32.0.23033.1002, dated 2026-03-09 — deliberately kept: newer Adrenalin builds are reported (community) to have a bug that empties/resets VRAM while it is used by an LLM workload, so I have not updated. The driver predates the bundled ROCm 10.2.0a20260930; if a specific driver version is known-good with this ROCm, please name it.
- **RAM:** 96 GB RAM, page file 96 GB (commit limit ≈ 192 GB, ~68 GB free at crash time)
- **TDR:** `TdrDelay=20`, `TdrDdiDelay=10` already set in `GraphicsDrivers` (the hang exceeds 20 s)
- **Model:** Qwen3.8-Flash-Next GSQ-RCO **IQ3_S** native pack, MTP rt, expert-profile
  - flags: `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 32768`, hipBLASLt table `gfx1201-hipblaslt-100500.txt`

## Steps to reproduce

The server restarts the engine each time (see timeline). Exact json:

```bash
{
 "exe": "J:\\Strata-main\\engine\\strata.exe",
 "args": [
  "--pack",
  "J:\\Strata-data\\packs\\iq3_s",
  "--native",
  "J:\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
  "--ple-gguf",
  "J:\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
  "--expert-profile",
  "J:\\Strata-main\\data\\expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--spec",
  "4",
  "--spec-min-p",
  "0.5",
  "--mtp",
  "J:\\Strata-data\\mtp\\rt",
  "--max-context",
  "229376",
  "--kv",
  "int8",
  "--kv-resident",
  "32768"
 ],
 "cwd": "J:\\Strata-main",
 "tokenizer": "J:\\Strata-data\\packs\\iq3_s\\tokenizer",
 "model_name": "qwen3.8-flash-next-iq3_s",
 "log": "J:\\Strata-main\\strata-iq3_s.log",
 "lib_dirs": [
  "J:\\Strata-main\\engine\\rocm\\bin"
 ],
 "port": 8080,
 "host": "0.0.0.0",
 "backend": "hip",
 "env": {
  "STRATA_HIPBLASLT_TUNING": "J:\\Strata-main\\tools\\hip\\gfx1201-hipblaslt-100500.txt"
 },
 "gpu": 1,
 "gpus_asked": true
}
```

## Symptom (engine log, 0.1.41, 262144)

The line `strata verify: capturing the N-token window` never prints — the engine dies before/at the first capture.

```text
strata generate: expert cache auto: 25.10 GiB free, 2560 MiB reserved (+218 MiB for the draft head) -> 9029 slots
strata generate: session is up (engine 0.1.41)
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2302 CUDA0 cache slots (4.38 GiB)
pref gemm: hipBLASLt tuning enabled (32 rows, gfx1201, version 100500)
strata verify: window up to 6 tokens, 75.8 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 2308 MiB of VRAM free with everything loaded
hipModuleUnload failed:
 error: unspecified launch failure
Exception Code: 0xC00000FD
```

Windows Error Reporting for the same minute:

- `LiveKernelEvent` **141** at 20:37:10 and 20:37:38 (dump `WATCHDOG-20261009-2037.dmp`)
- `APPCRASH`: `strata.exe`, faulting module `Dbghelp.dll` 10.0.19041.7725, exception `0xc0000005` (the engine's own crash handler faulting; the `0xC00000FD` in the log looks secondary)

## Timeline (2026-10-09, server auto-restarts the engine each time)

| Start | Engine | max-context | Outcome |
|---|---|---|---|
| 20:26:08 | 0.1.40.2 | 262144 | died at serve init; log printed its own warning: `447 MiB of VRAM free … that is little room for the verify windows' buffers under WDDM` (the #1275 shape, 700 MiB default reserve) |
| 20:32:43 | 0.1.41 | 262144 | 2308 MiB free → died at first capture, `unspecified launch failure` + `0xC00000FD` |
| 20:37:10 | 0.1.41 | 262144 | same; LiveKernelEvent 141 ×2 |
| 20:38:07 | 0.1.41 | 262144 | same (died within ~50 s) |
| 20:39:00 | 0.1.41 | 196608 | fine, served for hours |
| 21:05:58 | 0.1.41 | **229376** | **fine**: all six windows captured (`upload no error, sync no error`), then a real prompt of 119,136 fresh tokens read at 305 tok/s, a second 121,020-token prompt with 119,403 reused, decode 52 tok/s, expert-cache hit 92.6 %, KV streaming 96 % VRAM hit |

## What rules out the obvious suspects

- **Not VRAM pressure:** 2308 MiB free at the 262144 crash vs 2306 MiB free at the working 229376 start — identical headroom; the 0.1.41 2560 MiB auto-reserve (#1376) is active in both.
- **Not the commit limit (#1607/#1630):** ~68 GB of commit free at the crash time; the crash is at serve init, before any prompt.
- **Not TDR being too short:** `TdrDelay` is already 20 s; the GPU is wedged for more than that.
- **Not model/context infeasible on this card:** the docs' own R9700 runs IQ3_S at 431072 ctx on Linux, and #1565 ran IQ3_S @ 262144 on gfx1201 (ROCm). It's the Windows/WDDM capture path.


## Not yet tried (happy to run anything)

`--prefill auto:4096` at 262144 (expect little given the identical VRAM), `--vram-reserve-mib 3584`,  `STRATA_HIP_WMMA=1`, a newer AMD driver — held back for the VRAM-clearing bug described above.

## Related

#961 (WDDM verify window wedges the driver → 0x141 → TDR; there on NVIDIA), #1275 (the VRAM-tight variant, matches my first 0.1.40.2 start), #1630, #1644 (GPU faults on native IQ packs on Windows HIP).

## Question

If a specific AMD driver version is known-good with ROCm 10.2.0a20260930, please name it.

Mehr auf der Site

Links zu Install, Modellen, Releases.