#5718 [Engine] gfx1200 (RX 9060, Linux): "verify: timed out" (#267) during decode, not tied to spec decoding, RAM budget or pool workers
open · @Cyb3rb1ade · 9 コメント · GitHub で見る
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPModels & quantsLinux
本文
### Environment | | | |---|---| | Strata | 0.2.13.1, engine 0.1.59.1 (local HIP build via `update.sh`, gfx1200) | | GPU | AMD Radeon RX 9060 (gfx1200), 16 GB, **also drives the monitor** (KDE) | | ROCm / OS | rocm-core 7.2.4, CachyOS, kernel 7.3.1-1-cachyos | | CPU / RAM | Ryzen 5 8600G (6C/12T), 62 GiB RAM, zram swap | | Model | Qwen3.8-Flash-Next UD-IQ4_XS (native pack + PLE), MTP drafter | | Args | `--max-context 131072 --kv int8 --kv-resident 32768 --resident-budget-gib 38 --vram-reserve-mib 3072 --prefill auto --spec 4 --spec-min-p 0.5 --mtp`, `reasoning_budget_tokens` 32768 | ### What happens Mid-decode (thinking or writing a file via an agent), the token counter freezes for >10 s, then the watchdog stops the engine: ``` the engine stopped unexpectedly (exit code -15). Unrecoverable native verification failure: verify: timed out at layer 43; its GPU waits were released but the GPU did not finish within 5 s (#267) ``` Seen at layers 17, 21, 22, 27, 43, 47. Six times in about four hours today. The next request restarts the engine (about a minute) and the conversation cache is lost. ### Controlled reproduction (same prompt each time) Prompt: "write a command line calculator in Rust" (parser, functions, 8 unit tests), `reasoning_effort: high`, `temperature 0.6`, `max_tokens 40000`. | Variant | Result | |---|---| | default (above) | stall after 407 s / 7,998 thinking tokens (layer 22) | | `"mtp": false` (no draft layer, so no spec verify) | stall after 328 s / 5,357 tokens (layer 27) | | `--resident-budget-gib 30` | stall after 431 s (layer 21) | | `--pool-workers 4` (default is all 6 cores) | stall after 252 s / 4,768 tokens (layer 17) | 4 of 4 crashed. So it is **not** tied to speculative decoding (the message says "verify" even with the drafter off), to the expert RAM budget or to the CPU worker count. It is also not deterministic: in a pi agent session about 10 requests ran fine (including one with 5,700 tokens in 263 s) before the 14:18 crash at layer 43. ### What I checked - VRAM free at window capture: about 3.0 to 3.2 GiB (reserve 3072 MiB), so not the low-reserve case. - `MemAvailable` never below 23 GB during the 30 GiB run. Engine RSS is 50 to 57 GiB (includes mapped GGUF pages). - Kernel log: no amdgpu/ring/timeout lines. There are `page allocation failure` lines from other processes (Firefox, ktailctl, claude-desktop), but they do not line up with every stall. - Decode speed 20 to 39 tok/s, about 18 to 19 without the drafter. ### Related #267 (closed), #4862 (verify timeout, 2x RX 9070 XT, Linux), #5649 (gfx1200 stalls on long prompts). This one is on gfx1200 during **decode**, not prompt reading. ### Question Is there a debug switch or log level that would show where the GPU wait hangs (which kernel/layer queue)? I can run more tests on request, for example with the display GPU unloaded or with the older engine builds.
関連リンク
インストール・モデル・リリースへのサイト内リンク。