#2729 [Bug]: gfx1100 (RX 7900 XT 20 GB, 31 GiB RAM): amdkfd SVM restore hangs and kswapd order-0 alloc failures persist with HSA_USERPTR_FOR_PAGED_MEM=0; suspend with the SVM mapping active never resumes

Description

## What happened

Four kernel-level events on one box, all in the amdkfd/SVM restore paths, on a 31 GiB RAM / 20 GB VRAM machine. Two of them are evidence the published workaround does **not** cover: `HSA_USERPTR_FOR_PAGED_MEM=0` removes the amdkfd hang, but not the reclaim pressure behind it, and suspend/resume with the SVM mapping active never returns.

This is a second machine for the mechanism in #1705 (that box had 125 GiB RAM and the env var ended everything there). We are filing a new issue that links it, per the closing note there, because the new evidence below is not in #1705. Related: #750, #920, and the compaction-stall docs section being added in #2413.

### Environment

- Engine **0.1.43.1**, HIP backend (`build-hip`, archs `gfx1100`), args `--resident-experts --vram-reserve-mib 3072 --kv int8`
- **AMD Radeon RX 7900 XT 20 GB (gfx1100)**, one GPU, also drives the desktop (Wayland)
- **CachyOS** (Arch), kernel `7.2.9-2-cachyos` PREEMPT, **31 GiB RAM**, zram swap, zswap off, model files on NVMe
- Qwen3.8-Flash-Next, 131072 ctx, expert pool 14.2-20.6 GiB held as one paged SVM region (`STRATA_RESIDENT_PIN=0`, per the ~32 GB box guidance in TROUBLESHOOTING #649)
- One long-lived engine process; expert pages migrate RAM<->GPU under normal inference load

### Events 1 and 2 - 2026-10-12, before the env var was set

**1. `12:38:28` - `kworker` in D state inside `svm_range_restore_work`, then the box died** (boot ended 12:38:47, manual reboot 12:42):

```text
Workqueue: events_freezable svm_range_restore_work [amdgpu]
Call Trace:
 ? walk_pgd_range+0x460/0xea0
 ? walk_page_range_mm_unsafe+0x149/0x330
 ? hmm_range_fault+0x78/0xb0
 ? amdgpu_hmm_range_get_pages+0xcb/0x160 [amdgpu]
 ? svm_range_validate_and_map+0x633/0x18c0 [amdgpu]
 ? svm_range_restore_work+0xe1/0x370 [amdgpu]
 ? process_scheduled_works+0x1fa/0x690
```

followed by `warn_alloc` / `kswapd0: page allocation failure: order:0`.

**2. `19:32:25` (also `13:03:10`, `15:48:59`) - hung task, engine unkillable until reboot:**

```text
INFO: task kworker/1:67:15296 blocked for more than 122 seconds.
Workqueue: events_freezable amdgpu_amdkfd_restore_userptr_worker [amdgpu]
Call Trace:
 __schedule+0x753/0x2270
 schedule+0x55/0x810
 schedule_preempt_disabled+0x30
 __mutex_lock+0x1e9/0x5d0
 amdgpu_amdkfd_restore_userptr_worker+0x82/0xad0 [amdgpu]
```

The mutex is held by `svm_range_restore_work` -> `amdgpu_hmm_range_get_pages` -> `hmm_range_fault` -> `filemap_fault` -> `folio_wait_bit`: **the GPU's SVM expert restore waits inside a model-file page fault.** At user level that is the prompt watchdog plus a 60-162 s freeze, then SIGABRT:

```text
reading the prompt (batched): waiting for the GPU (attention, router) at layer 21
host idle 242 s, 7/7 workers parked, 24/24 jobs done, 0 threads on disk
no progress for 60 s during a request (issue #29)
```

The engine then exits `-6` and the server restarts it. ~10 stalls/day on the Q2_0 tier.

### Event 3 - 22:46:26, suspend with the engine loaded never resumed

```text
Oct 12 22:46:26 systemd-logind[697]: suspend requested from client PID 1531 ('org_kde_powerde')
Oct 12 22:46:26 kernel: PM: suspend entry (deep)
```

Nothing after - no resume line, no amdgpu line. The box needed a hard power cycle (reboot 22:51:35). Third time suspend/resume with the Strata SVM mapping active has failed to come back, same class as event 1. **This one is independent of the env var**, so we stop the engine before suspend via a system-sleep hook.

### Event 4 - 2026-10-13 12:53:52, WITH the workaround set

`HSA_USERPTR_FOR_PAGED_MEM=0` and `GPU_PINNED_MIN_XFER_SIZE=1048576` were both in the engine's `env` block for this run, and the reclaim pressure is still there:

```text
Oct 13 12:53:52 kernel: kswapd0: page allocation failure: order:0,
  mode:0xc0de0(GFP_KERNEL|__GFP_HIGH|__GFP_ZERO|__GFP_COMP|__GFP_NOMEMALLOC)
Oct 13 12:53:52 kernel: PID: 141 Comm: kswapd0 Tainted: G U 7.2.9-2-cachyos
  warn_alloc <- __alloc_pages_slowpath <- __alloc_frozen_pages_noprof
  <- folio_alloc <- isolate_lock_cluster <- cluster_alloc_swap_entry
  <- folio_alloc_swap <- shrink_folio_list <- evict_folios <- kswapd
```

Same boot: coder tier (IQ1_M, expert pool 17.14 GiB, `STRATA_RESIDENT_HEADROOM_GIB=8`, `--kv int8`), **0 file-tier expert reads**, MemAvailable 10 GiB, swap 0 used. 31 order-0 allocation failures in that boot. So on this box the env var removes the amdkfd hang (events 1-2 have not recurred with it set) but not the RAM side of the same subsystem.

### What we would like upstream

1. A note in the `docs/AMD_HIP.md` reclaim section that on a ~31 GiB box the env var ends the *hang* but not the order-0 reclaim failures, and that suspend with the SVM mapping active is a separate failure class - so people do not read "set the env var" as a complete fix.
2. If it is cheap: a load-time warning when the resident expert complement plus headroom exceeds usable RAM, which is exactly the condition that produced events 1, 2 and 4 here.
3. Nothing here is a request for new engine behaviour on the GPU side - the traces are kernel-side, and our box is now stable on the coder tier (0 file reads, 0 stalls).

### Notes for triage

- All four traces are in the SVM/amdkfd restore paths (`svm_range_restore_work`, `amdgpu_amdkfd_restore_userptr_worker`) - one subsystem.
- The paged pool is one large SVM region (tens of GiB); page-ins/outs happen under normal inference load, not only during fault storms.
- Bisect hint: events 1-2 predate `HSA_USERPTR_FOR_PAGED_MEM=0`; event 3 is the suspend path and is independent of it; event 4 is with it set.
- We are on 0.1.43.1, not 0.1.44 - happy to re-test on 0.1.44 and reopen/comment here if any of the four classes still reproduces.