#3021 [Bug]: gfx1100 2x RX 7900 XT layer split + --resident-experts (0.1.45,Q2_0):16K–100K prompt prefill stalls (watchdog #29), GPU reset / device lost

Descrição

### What happened

Running Qwen3.8-Flash-Next Q2_0 on two RX 7900 XT cards with `--resident-experts` and a layer split. Short prompts work; **every long prompt I tried hangs during prompt prefill** (16K, 40K and 100K tokens). Two outcomes:

1. The prompt watchdog fires and the engine stops itself:
   `no progress for 60 s ... waiting for the GPU (attention, router) at layer N of the prompt chunk from token M` (issue #29).
2. The kernel then tries to reset the GPU and fails: `MES failed to respond ...` -> `GPU reset begin!` -> `device lost from bus!` -> `GPU reset end with ret = -19` / `GPU Recovery Failed: -19`. Both cards can end up in this state; only a host reboot recovers them.

The same model and engine on **one** card are stable at 128K, including a 100K-token prompt.

### Strata version, GPU, OS

- Engine **0.1.45** (HIP backend)
- 2x AMD Radeon RX 7900 XT (Navi 31, gfx1100, 20 GB each); an RTX 4060 Ti is present but not used by Strata
- Debian 13, kernel **6.12.113+deb13-amd64**, ROCm **7.1.1**, HIP 7.1.52802
- Container: `rocm/pytorch:rocm10.1.0_ubuntu24.04_py3.12_pytorch_release_2.14.0`
- CPU/RAM: Ryzen 5 7500F, 30 GiB RAM, 178 GiB swap
- Model: Qwen3.8-Flash-Next Q2_0 (62 GB model + 38 GB expert pack + 6.5 GB MTP), `--resident-experts`

### Exact flags (conservative run that still failed)

```text
--serve --pack <.../packs/q2_0> --native <...Q2_0-00001-of-00002.gguf>
--ple-gguf <...Q2_0-00002-of-00002.gguf>
--expert-profile <.../data/expert-profile.bin> --expert-cache auto
--prefill 4096 --pcie-frac 0 --spec 4 --spec-min-p 0.5 --mtp <.../mtp/rt>
--max-context 65536 --vram-reserve-mib 1024 --kv int8 --resident-experts
--layer-split auto
```

Config: `"gpu": [0, 1]`, `"layer_split": "auto"`, `"backend": "hip"`,
`"STRATA_RESIDENT_PIN": "0"`, `"HSA_USERPTR_FOR_PAGED_MEM": "0"`.
The engine started with layers 0-24 on CUDA0 and 25-47 on CUDA1; the two caches
held 11606 and 10550 experts; 6.65 GiB were resident in RAM (pageable); 798 MiB
(CUDA0) and 846 MiB (CUDA1) of VRAM were free.

Two earlier runs used the same setup without the conservative flags:
`--prefill auto`, no `--pcie-frac`, no `--vram-reserve-mib`, `--max-context 131072`.
The first of those had 21894 cached experts (11503 + 10391, ~99.7% of the routed
mass), 10.00 GiB resident, and 474/522 MiB free.

### Reproduction

1. Start the server with the config above.
2. Send a short prompt: it answers normally.
3. Send a prompt of ~16K tokens (the conservative run) or ~40K/100K (the 128K runs).
4. At some point during prefill the engine stops making progress. After 60 s the
   watchdog aborts it; in the 128K runs the kernel then fails to reset the GPU.

I used generated filler text (`"the " * N`) with a short question at the end, sent
through `/v1/chat/completions`; nothing else was running on the box.

### Logs

Full engine log and server log are available (63 KB `strata-q2_0.log`, 2 KB
`server.out`); the key lines are below. dmesg timestamps are UTC.

**Conservative run (64K, 16K-token prompt):**

```text
strata serve: no progress for 60 s during a request (reading the prompt (batched):
waiting for the GPU (attention, router) at layer 29 of the prompt chunk from token 0)
- stopping the engine so the server starts it again (issue #29)
```

**128K run, 40K-token prompt (memlock raised to unlimited):**

```text
strata serve: no progress for 60 s during a request (reading the prompt (batched):
waiting for the GPU (attention, router) at layer 12 of the prompt chunk from token 16384)
- stopping the engine so the server starts it again (issue #29)
```

**Kernel, 128K run with a 100K-token prompt:**

```text
amdgpu: init_user_pages: Failed to get user pages: -1   (846 times at engine start)
[gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:126)
  in process strata pid ... thread strata pid ...
  in page starting at address 0x00007fcb300be000 from client 10
GCVM_L2_PROTECTION_FAULT_STATUS:0x00801031
  Faulty UTCL2 client ID: TCP (0x8)
  PERMISSION_FAULTS: 0x3
MES failed to respond to msg=SUSPEND
failed to suspend all gangs
MES might be in unrecoverable state, issue a GPU reset
GPU reset begin!
device lost from bus!
GPU reset end with ret = -19
GPU Recovery Failed: -19
```

The engine printed the matching userspace error:

```text
Memory access fault by GPU node-1 ... on address 0x7fcb300be000.
Reason: Page not present or supervisor privilege.
```

**Kernel, 128K run, 40K-token prompt** (after `RLIMIT_MEMLOCK` was raised to
`unlimited`; `init_user_pages` still failed 1814 times at start):

```text
SMU: response:0xFFFFFFFF ... message:TransferTableSmu2Dram?
Failed to export SMU metrics table!
MES failed to respond to msg=REMOVE_QUEUE
failed to remove hardware queue from MES, doorbell=0x1008
MES might be in unrecoverable state, issue a GPU reset
GPU reset begin! / device lost from bus!   (14 reset + device-lost pairs)
```

### What I tried / ruled out

- **P2P:** `rocm-smi --showtopoaccess` reports `GPU0 -> GPU1 = False`; RDNA3
  consumer cards do not expose P2P here. The cards are behind a Broadcom
  PEX880xx PCIe Gen4 switch sharing one x16 uplink, so card-to-card traffic
  also has to go through the host. The engine's link probe reports
  `CUDA0 link -1.0 GB/s -> miss cost x1.00` (same for CUDA1).
- **memlock:** raised the server process to `RLIMIT_MEMLOCK=unlimited`; the
  watchdog/GPU failure persisted and `init_user_pages` kept failing.
- **PCIe pressure:** `--pcie-frac 0` (no PCIe expert fetches) did not help.
- **Prefill chunk:** `--prefill 4096` did not help; the stall moved from
  layer 12/token 16384 to layer 29/token 0.
- **VRAM pressure:** `--vram-reserve-mib 1024` left 798/846 MiB free; still hung.
- **Host-page reclaim workaround:** `HSA_USERPTR_FOR_PAGED_MEM=0` (as in #1705)
  did not help on this machine.

### Single-GPU baseline (same model, same engine)

- 128K context, one card: a 100,078-token prompt is accepted and answered
  correctly; prefill 92.3 s (1084 tok/s); decode 69.7-72.8 tok/s; a repeat of
  the same prompt hits the prompt cache (100,073 tokens reused, 0.96 s).
- Short chat: ~49.8 tok/s end to end.

So the Q2_0 pack, tokenizer, MTP, 128K context and the single-GPU path are fine.

### Related issues

- #1705 (closed): gfx1100 prefill "no progress" from host-page reclaim;
  `HSA_USERPTR_FOR_PAGED_MEM=0` fixed it there. In a comment, HackityAnn reports
  the same class on 2x RX 9070 XT with `--resident-experts` + layer split where
  HSA did not help, but **`GPU_PINNED_MIN_XFER_SIZE=1048576` did**.
- #1407 (closed): gfx1200 2x RX 9060 XT layer split prompt stalls (watchdog #29).
  A comment traces it to GFXOFF + SDMA adaptive expert swaps and reports
  **`STRATA_HIP_ADAPT_KERNEL_COPY=1`** and **disabling GFXOFF** as fixes.
- #2637 (closed): gfx1100 2x 7900 XTX, `--pipeline-windows 2` + MTP, exits on an
  18K prompt; fixed by `STRATA_MTP_STAGE0=1`. My config does not use
  `--pipeline-windows`, but the hardware and prompt length are close.

### Question

Is this the same class as #1705 / #1407? Before I keep experimenting, would you
recommend trying `GPU_PINNED_MIN_XFER_SIZE=1048576`,
`STRATA_HIP_ADAPT_KERNEL_COPY=1`, and/or disabling GFXOFF on 0.1.45? I can run
them one at a time and report back with the engine log and dmesg for each arm.

I am currently back on one card at 128K (stable). I can attach the full
`strata-q2_0.log`, `server.out` and the dmesg excerpts from the three runs.