#5718 [Engine] gfx1200 (RX 9060, Linux): "verify: timed out" (#267) during decode, not tied to spec decoding, RAM budget or pool workers

open · @Cyb3rb1ade · 9 comentarios · En GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPModels & quantsLinux

Descripción

### Environment

| | |
|---|---|
| Strata | 0.2.13.1, engine 0.1.59.1 (local HIP build via `update.sh`, gfx1200) |
| GPU | AMD Radeon RX 9060 (gfx1200), 16 GB, **also drives the monitor** (KDE) |
| ROCm / OS | rocm-core 7.2.4, CachyOS, kernel 7.3.1-1-cachyos |
| CPU / RAM | Ryzen 5 8600G (6C/12T), 62 GiB RAM, zram swap |
| Model | Qwen3.8-Flash-Next UD-IQ4_XS (native pack + PLE), MTP drafter |
| Args | `--max-context 131072 --kv int8 --kv-resident 32768 --resident-budget-gib 38 --vram-reserve-mib 3072 --prefill auto --spec 4 --spec-min-p 0.5 --mtp`, `reasoning_budget_tokens` 32768 |

### What happens

Mid-decode (thinking or writing a file via an agent), the token counter freezes for >10 s, then the watchdog stops the engine:

```
the engine stopped unexpectedly (exit code -15). Unrecoverable native verification failure:
verify: timed out at layer 43; its GPU waits were released but the GPU did not finish within 5 s (#267)
```

Seen at layers 17, 21, 22, 27, 43, 47. Six times in about four hours today. The next request restarts the engine (about a minute) and the conversation cache is lost.

### Controlled reproduction (same prompt each time)

Prompt: "write a command line calculator in Rust" (parser, functions, 8 unit tests), `reasoning_effort: high`, `temperature 0.6`, `max_tokens 40000`.

| Variant | Result |
|---|---|
| default (above) | stall after 407 s / 7,998 thinking tokens (layer 22) |
| `"mtp": false` (no draft layer, so no spec verify) | stall after 328 s / 5,357 tokens (layer 27) |
| `--resident-budget-gib 30` | stall after 431 s (layer 21) |
| `--pool-workers 4` (default is all 6 cores) | stall after 252 s / 4,768 tokens (layer 17) |

4 of 4 crashed. So it is **not** tied to speculative decoding (the message says "verify" even with the drafter off), to the expert RAM budget or to the CPU worker count. It is also not deterministic: in a pi agent session about 10 requests ran fine (including one with 5,700 tokens in 263 s) before the 14:18 crash at layer 43.

### What I checked

- VRAM free at window capture: about 3.0 to 3.2 GiB (reserve 3072 MiB), so not the low-reserve case.
- `MemAvailable` never below 23 GB during the 30 GiB run. Engine RSS is 50 to 57 GiB (includes mapped GGUF pages).
- Kernel log: no amdgpu/ring/timeout lines. There are `page allocation failure` lines from other processes (Firefox, ktailctl, claude-desktop), but they do not line up with every stall.
- Decode speed 20 to 39 tok/s, about 18 to 19 without the drafter.

### Related

#267 (closed), #4862 (verify timeout, 2x RX 9070 XT, Linux), #5649 (gfx1200 stalls on long prompts). This one is on gfx1200 during **decode**, not prompt reading.

### Question

Is there a debug switch or log level that would show where the GPU wait hangs (which kernel/layer queue)? I can run more tests on request, for example with the display GPU unloaded or with the older engine builds.

En el sitio

Enlaces a install, modelos, releases.