#3443 [Engine] gfx1201 (R9700), 0.1.45: watchdog self-stop at the layer-48 completion handshake — GPU rang 48, pool idle, 9 GiB of the engine in swap (class A of #1705, signature of #2049)

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

Engine 0.1.45, Linux/HIP, gfx1201 (Radeon AI PRO R9700 32 GB): the #29 watchdog stopped the engine with `no progress for 60 s during a request (request -1)`. The stall report shows the device had finished the whole window and rung all 48 layers, the expert pool had nothing in flight, and the host process was 9 GiB deep in swap. The process died by `abort()` (SIGABRT, core dumped), and the server restarted the engine 2 min 40 s later.

This is not a new signature: it is the layer-48 completion handshake of #2049 and the class-A stall of #1705. Both are closed for age, and `docs/DETAILS.md` asks for the stall report to be filed, so this is the same stall on a third platform (Linux/HIP, gfx1201) and a newer engine.

How I read it:

- `layer step 48` is one past the last block (this pack has 48), i.e. the completion handshake after the last layer, not a decode step. Same site as #2049.
- The GPU is alive and done: `served 48, plan (A) 48, copies (B) 48`, and when the abort path called `release_gpu_waits()` the device answered in 26 ms. The host side is what never moved.
- The pool is quiescent: 48 of 48 jobs done, `epoch == batch epoch`, all 15 workers asleep, host idle 66.7 s.
- `0 of 41` threads in state D, so it is not a visible blocked read, and the #1407 line (`no step finished for %d s, but the file tier is still being read`) never appears in the log.
- Stage `request -1` = between requests. The line immediately before the stall was a park, and a slow one: `parked 156419 tokens in 5609.5 ms`, far outside the ~760 ms median for 150-210K parks in #2049's table. The park completed and logged normally; the stage that stopped is `request -1`.

Prior occurrences on the same box, same signature: two more on 0.1.41, 2026-10-10 (restarts 15:23:59 and 15:34:48, so aborts at ~15:22 and ~15:33, 11 minutes apart, the cascade shape of #2049). Both stopped in the prompt read with the identical handshake state, `host at layer step 48; the GPU rang 48`, pool idle 78.9 s, `memory: 58030 MiB resident, 3290 MiB in swap, 4519 MiB RAM available`. Three aborts, three engines, identical handshake state.

Ruled out:

- #2065 VRAM floor: `VRAM free when this request began: 2660 MiB`, well above the 512 MiB warning threshold, and the LOW tag did not print.
- #1407 slow file tier: the line never appears; `0 of 41` threads in state D.
- The park path from 0.1.43.1: the park itself completed and logged; the stage that stopped is `request -1`.
- A decode/NaN problem: `non-finite logits` never appears.
- No KFD trace (unlike #1705): this account cannot read the kernel journal, so I cannot show `Evicting process` / `Restoring` pairs. The reclaim evidence is the process's own swap footprint plus the idle-GPU handshake state.

Questions:

1. Is this the same KFD userptr / host-reclaim class as #750, #920 and #1705 on 0.1.45, or does a completed-but-unreaped window with an idle pool point somewhere else?
2. On gfx1201 with a buffered expert arena (no `--mmap-experts`), which workaround first: `HSA_USERPTR_FOR_PAGED_MEM=0` (reported as not effective on gfx1201 by HackityAnn in #1705) or `GPU_PINNED_MIN_XFER_SIZE=1048576`?
3. Should the watchdog treat *GPU rang all 48 layers with the pool idle* as progress and re-issue the wait once, instead of aborting? The forced release finished the window in 26 ms, so the work was already done and a restart costs a full 33 GiB model reload.
4. Is there a supported way to make the park/handshake copy report progress for stage `request -1`, the way 0.1.43.1 did for the named park stage?

I will A/B the chosen env variable with the same workload and paired timings and report back either way.

### Smallest steps or command that reproduce it

Not reproducible on demand. What the three occurrences have in common: a long session at large context with `--batch 2`, two live slots (78K and 156K tokens in the 2026-10-20 event), plus a multi-GB parked conversation, on a box where the system is already 60+ GiB into swap. The last occurrence began after the park line quoted above; the crash itself is at the next handshake.

### Reproduction status

Reproduced locally

### Strata version

engine 0.1.45, local HIP build (`BUILD.json`: source `local-hip`, archs `[gfx1201]`, src `b95137167bd9b751`); fork merge of upstream tag v0.1.45 (`74f5671a`)

### Graphics card, memory and system

Radeon AI PRO R9700 32 GB (gfx1201) · 125.7 GiB RAM, 96 GiB swap partition (61-64 GiB in use at the abort), `vm.swappiness=10`, no zram · Debian 13, kernel 6.12.107+deb13-amd64 · ROCm 7.10.0a20251120 (TheRock pip wheels)

### Model, size and settings

Qwen3.8-Flash-Next GSQ-RCO IQ2_XS native pack (68 GB, 48 blocks, 33.02 GiB experts loaded, 15932 cache slots) · 262144 context · `--kv int8 --kv-resident 32768` · pictures on · `--spec 4 --spec-min-p 0.5 --mtp <rt> --batch-mtp` · `--batch 2` · `--conversation-cache-mib 8192` · `--prefill auto:32768` · `--vram-reserve-mib 3072` · env `STRATA_HIP_WMMA=1`, `STRATA_HIPBLASLT_TUNING=<gfx1201 100200 table>`; `HSA_USERPTR_FOR_PAGED_MEM` and `GPU_PINNED_MIN_XFER_SIZE` are NOT set · watchdog defaults (`STRATA_WATCHDOG_S=60`, `STRATA_WATCHDOG_IO_S=600`)

### Logs

Verbatim from `strata-iq2_xs.log`, 2026-10-20 16:37:47 EEST (UTC+3):

```text
strata serve: no progress for 60 s during a request (request -1) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.45): stage "request -1" for 59 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  VRAM free when this request began: 2660 MiB
  expert pool: epoch 25379158, batch epoch 25379158: 48 of 48 jobs claimed, 48 done; 15 of 15 workers parked, 15 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping; host idle for 66654 ms
  verify window (last window, not the current stage): 6 tokens at position 156872, host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48
  threads waiting on the disk (state D): 0 of 41
  memory: 35981 MiB resident, 9068 MiB in swap, 3773 MiB RAM available; 4751 major page faults so far
  2 s later:
  expert pool: epoch 25379158, batch epoch 25379158: 48 of 48 jobs claimed, 48 done; 15 of 15 workers parked, 15 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping w13=sleeping w14=sleeping; host idle for 68675 ms
  verify window (last window, not the current stage): 6 tokens at position 156872, host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48
  threads waiting on the disk (state D): 0 of 41
  memory: 35098 MiB resident, 9878 MiB in swap, 3408 MiB RAM available; 4751 major page faults so far
strata: released the verify window's GPU waits (#267): the GPU finished in 26 ms
```

`coredumpctl info 213777`:

```text
Signal: 6 (ABRT)
PID: 213777 (strata)
Executable: .../llm-engine/strata/engine/strata
Unit: [email protected], User Unit: [email protected]
```

The server logged `the engine had stopped (exit code -6)` at 16:39:55 and restarted it at 16:39:59; serving again at 16:40:30. The 30 GB core is still on disk. The full log excerpt with all three aborts, and the two 2026-10-10 stall reports, can be attached if useful.

---

*Authorship disclosure: this report was assembled by the operator's agent from the engine log and `coredumpctl` on the machine that stalled. Every quoted line is verbatim from `strata-iq2_xs.log`; the interpretation is mine and is labelled as such where it goes beyond the log.*