#5614 [Engine] RX 9070 XT gfx1201: 0.1.58 verify unspecified launch failure triggers MES REMOVE_QUEUE GPU reset
open · @tomsun66-star · 4 comentários · No GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPModels & quantsLinux
Descrição
## Problem Strata engine `0.1.58` on an RX 9070 XT (`gfx1201`, HIP, Linux) repeatedly ends with: ```text strata serve: fatal GPU error (verify: unspecified launch failure): the engine ends so that the server can start it again ``` The kernel then logs a GPU reset on the same card: ```text amdgpu 0000:2f:00.0: MES(0) failed to respond to msg=REMOVE_QUEUE amdgpu 0000:2f:00.0: failed to remove queue from MES, doorbell=0x1002 amdgpu 0000:2f:00.0: MES might be in unrecoverable state, issue a GPU reset amdgpu 0000:2f:00.0: GPU reset succeeded, trying to resume amdgpu 0000:2f:00.0: VRAM is lost due to GPU reset! ``` The desktop session was affected: ```text gnome-shell: amdgpu: The CS has cancelled because the context is lost. This context is innocent. ``` ## Environment - OS: Ubuntu 26.04.1 LTS - Kernel: `7.3.0-10-generic` - CPU: AMD Ryzen 7 5800X3D - RAM: 96 GiB total - GPU: PowerColor Radeon RX 9070 XT 16 GB, PCI `0000:2f:00.0`, driver `amdgpu` - Engine: Strata `0.1.58`, `BUILD.json` variant `rx9070`, archs `gfx1200`, `gfx1201` - Model: `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S` - Server config: `model_name qwen3.8-flash-next-iq3_s`, `backend hip`, `gpu 1`, `port 8090` - Engine args, sanitized: ```text --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <mtp-path> --max-context 163840 --kv int8 --kv-resident 65536 --vram-reserve-mib 3072 --ple-io mmap ``` - Start script exported: ```text HSA_ENABLE_SDMA=0 GPU_MAX_HW_QUEUES=4 STRATA_HIP_WMMA=0 STRATA_PF_FUSED=0 ``` Caveat: the engine log says matrix-core prompt attention is on by default on `gfx1201`; the log does not explicitly confirm whether the exported `STRATA_HIP_WMMA=0` reached the engine. ## Reproduction status Reproduced locally, but not reduced to a minimal deterministic reproduction. Observed sequence: 1. Start Strata server with the above config. 2. Use normal opencode agent traffic with long prompts. 3. Engine logs successful long prompts, including: ```text strata serve: prompt 114892 tokens = 90109 reused + 24783 read in 16280 ms (1522.3 tok/s), 139 generated in 2455 ms (56.6 tok/s), drafts accepted 84 of 91, 6 checkpoints ``` 4. Engine logs: ```text strata serve: fatal GPU error (verify: unspecified launch failure): the engine ends so that the server can start it again ``` 5. Server restarts engine. 6. After restart, a short `prompt 54 tokens` request is logged, then the same fatal GPU error appears again. 7. Kernel logs MES `REMOVE_QUEUE` failure and GPU reset. ## Expected behavior The engine should either complete the request or fail the request without causing a kernel-level MES wedge, GPU reset, VRAM loss, or desktop crash. ## Actual behavior - Engine exits with fatal GPU error. - Start script logs engine exit code `3` for the fatal GPU error. - Kernel performs a GPU reset. - VRAM is lost. - Desktop session loses GPU context. ## Additional local observation: HTTP health can look healthy while the engine is not loaded At `22:13:44`, the local start script logged: ```text Strata :8090 laeuft bereits (unbekannt) ``` This means `/health` answered, but `/v1/models` did not report the expected model. At `22:14:17`, the old engine log shows: ```text Hip error: 'unspecified launch failure'(719) at .../hipblaslt.cpp:204 hipModuleUnload failed: error: unspecified launch failure ``` Then a new engine start. No kernel GPU reset was logged in the `22:13:00` to `22:15:30` window. Interpretation, not proven: the “start script says running, but GPU monitoring shows no VRAM” situation is consistent with the HTTP server being alive while the engine was not loaded or had lost GPU state. ## Related cases and differences | Issue | What it reports | Similarity | Difference from this case | |---|---|---|---| | #5608 | RX 9070 XT gfx1201, IQ3_S, engine 0.1.57/0.1.58, repeatable GPU page faults, exit `-13` | Same GPU, same model family, same engine generation | Kernel shows MES `REMOVE_QUEUE` / GPU reset, not gfxhub page faults; engine exit code was `3` | | #5559 | Engine 0.1.58 crashes repeatedly with GPF in `libhsa-runtime64`, exit `-6/-11`, under 35K-92K prompts; 0.1.57 clean | Same GPU family, same engine `0.1.58`, long prompts | Kernel shows no GPF in `libhsa-runtime64`; fatal string is `verify: unspecified launch failure` plus MES reset | | #5103 | Our own prior report: RX 9070 XT MES wedge on 20K+ token prefill, silent whole-host lockups, engine 0.1.54 build 2026-10-30 | Same hardware/user, MES wedge, high-context prefill | Engine is `0.1.54`, not current `0.1.58`; current visible symptom includes `verify: unspecified launch failure` and desktop crash after GPU reset | | #5549 | Benchmark single vs dual RX 9070, engine 0.1.58, UD-IQ4_XS | Same GPU family and engine version | Not a crash report; only benchmark | GitHub search context: - 43 issues contain `unspecified launch failure`. - 98 issues mention `RX 9070 XT`. - No exact match was found for the combined signature: `verify: unspecified launch failure` + MES `REMOVE_QUEUE` + GPU reset + desktop crash. ## Important caveat The old engine `0.1.44.1` also logged: ```text Hip error: 'unspecified launch failure'(719) at .../hipblaslt.cpp:204 hipModuleUnload failed: error: unspecified launch failure ``` and restarted at `22:14:17`. In that old-engine instance, no kernel GPU reset was logged. Therefore this report should not claim that only `0.1.58` ever sees `unspecified launch failure`. It should claim that `0.1.58` produced the fatal GPU error plus kernel MES reset/desktop crash in the observed sequence. ## Not tested - Engine `0.1.57` on this machine. - `0.1.58` with confirmed `STRATA_HIP_WMMA=0` and `STRATA_PF_FUSED=0` reaching the engine. - `STRATA_GFX12_DEFAULTS=0`. - Disabling MTP / `--spec 4` / draft layer. - `AMD_SERIALIZE_KERNEL` or other AMD diagnostic variables. - Whether the old engine’s `hipblaslt` unspecified launch failure can also cause a desktop crash.
No site
Links install, modelos, releases.