#2850 [Bug]: HIP/Linux, 2x RX 7900 XTX: the page-locked RAM complement takes 2-6 min on many starts (median 300 s in an interleaved A/B: ROCr's KFD userptr allocation returns EAGAIN thousands of times); with THP disabled for the process, 34 s

说明

### What happened

With `--resident-budget-gib 40` on two RX 7900 XTX (layer split), the engine page-locks a 28.47 GiB cache complement
with one `cudaHostAlloc(Mapped | Portable)` (`FileExpertSource::pin_cache_complement`). On this machine that step
often takes 2-6 minutes, at other times 25-35 s, with the same config. Strata's own copy into the complement is 14-17 s every
time. The rest of the time goes to one call into the driver that fails and is retried thousands of times.

With transparent huge pages (THP) disabled for the server process (`prctl(PR_SET_THP_DISABLE)`, inherited by the
engine), the same start took 33-35 s in 4 of 4 starts, against 294-322 s for the default in 4 of 4 starts,
interleaved, same config, same hour. Decode and prompt speed did not change measurably.

#### Interleaved A/B, real engine (2026-10-16, 15:39-16:27)

Same config each time (below), back-to-back fresh starts, nothing else on the GPUs; the bench after each start is one
short prompt and one 32K prompt, 512 tokens generated at the server's default sampling (temperature 1.0), so the
tok/s columns are single requests. "alloc" = the complement total minus its copy: the allocation plus a few seconds of
bookkeeping.

| start | THP | complement total (s) | copy (s) | alloc (s) | load to ready (s) | gen short | gen 32K | prompt 32K |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| d1 | default | 300 | 14 | 286 | 370 | 68.0 | 70.5 | 1955 |
| n1 | off | 35 | 15 | 20 | 77 | 67.4 | 68.2 | 1912 |
| d2 | default | 301 | 14 | 287 | 363 | 67.7 | 61.8 | 1570 |
| n2 | off | 33 | 15 | 18 | 75 | 40.0 | 72.2 | 1902 |
| d3 | default | 294 | 14 | 280 | 356 | 77.6 | 71.4 | 1812 |
| n3 | off | 33 | 15 | 18 | 73 | 74.8 | 80.9 | 1918 |
| d4 | default | 322 | 15 | 307 | 370 | 67.5 | 65.6 | 1900 |
| n4 | off | 35 | 17 | 18 | 74 | 67.1 | 72.6 | 1972 |
| **median, default (4)** | | **300.5** | 14 | **286.5** | **366.5** | 67.8 | 68.0 | 1856 |
| **median, THP off (4)** | | **34.0** | 15 | **18.0** | **74.5** | 67.2 | 72.4 | 1915 |

Two more default starts were run under strace (`strace -f --seccomp-bpf -T -tt -e trace=ioctl`): 91 s (its copy took
73 s that time, the allocation 8.5 s with 93 retries) and 223 s (below). Before the test, in normal use the same config
started 8 times on 2026-10-16 with a complement time of 174 (that one took the paced #1250 path), 33, 132, 376, 347, 316, 38 and 386 s. Over 76 starts in
our logs since 2026-10-12 (0.1.43.1 and 0.1.44.1) the median is 27 s and the slow ones are 75-497 s: many
calibration starts in a row were fast; in normal use (a coding benchmark between starts) most are slow.

#### Where the time goes

ROCr implements this `hipHostMalloc` as an anonymous `mmap` of the whole size, `madvise(MADV_HUGEPAGE)` on it, then one
`AMDKFD_IOC_ALLOC_MEMORY_OF_GPU` (a KFD userptr: `/proc/<pid>/status` shows `VmLck` and `VmPin` 0 after it). From the
standalone test's trace:

```
mmap(NULL, 30571532288, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7c4843cb7000
mmap(0x7c4843e00000, 30569431040, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7c4843e00000
madvise(0x7c4843e00000, 30569431040, MADV_HUGEPAGE) = 0
ioctl(3, AMDKFD_IOC_ALLOC_MEMORY_OF_GPU, 0x7ffedcee90e0) = -1 EAGAIN (Resource temporarily unavailable) <2.336246>
ioctl(3, AMDKFD_IOC_ALLOC_MEMORY_OF_GPU, 0x7ffedcee90e0) = -1 EAGAIN (Resource temporarily unavailable) <0.063404>
... (the same call again, thousands of times)
```

In the real engine's 223 s start, the complement's allocation call was made **about 3,770 times from 16:20:09.8 to
16:23:28.4 (198.6 s)**, every attempt but the last returning EAGAIN; the last succeeded (0.34 s) and the GPU mapping after it took
0.29 s. In a slow start in normal use (sampled from `/proc`, no strace), one engine thread
was at 100% CPU for 5 minutes, all of it system time (`wchan` 0), while the process's resident memory grew by about
95 MB/s; the 28 GiB landed in the last ~20 s.

#### Standalone reproduction (no model, no kernels)

`pintest.cpp` (below) makes the same call (`hipHostMalloc(28.47 GiB, Mapped | Portable)`) and times it, with
`/proc/vmstat` deltas over the allocation:

| case | alloc | notes |
|---|---:|---|
| right after a real engine exited (item 1067, ~30 min of serving), THP default, under strace | **1071 s** | ~11,000 EAGAIN (1,069 s inside the failing calls); `pgfault` +31.2M for 7.46M pages; `thp_fault_alloc` +14,698; `compact_stall` +1,567; `pswpin` +257,614 |
| same call under the same strace, later | 2.9 s | 0 EAGAIN (so strace is not what makes it slow) |
| THP default, idle machine (several runs) | 2.9-3.2 s | `pgfault` = pages, compaction ~0 |
| THP default, page cache full of a 30 GB file | 6.9 s | `compact_stall` +2,538 |
| THP off (`PR_SET_THP_DISABLE`), idle / cache full / cache dropped / after a 28 GiB pinned process exits | 8.2 s each | no compaction at all |
| `malloc` + `memset(0)` + `hipHostRegister(Mapped \| Portable)` | 9.2 s | the compiler made it a calloc (memset 0.00 s), so the pages were faulted inside `hipHostRegister` |
| 15 pieces of 2 GiB | 3.6 s | |
| one visible GPU (`HIP_VISIBLE_DEVICES=0`) | 3.1 s | |

Only the first row hit the slow state. I could not bring it on with a standalone recipe (a full page cache, a dropped
cache, a large pinned process that just exited). Today's back-to-back real starts with the default were slow 5 of 6
times (the A/B's 4, the 223 s traced one; the other traced one was 91 s), but in a calibration run the night before,
many back-to-back starts of the same config were fast; I do not know what differs. So the
standalone rows other than the first show only the cost of each variant **in the fast state**. THP off is the only
variant measured in the slow state (the A/B above).

#### What I think happens (not confirmed: no root tracing on this machine)

The userptr allocation walks the range with HMM (`hmm_range_fault`) and returns `-EAGAIN` when the range is
invalidated during the walk; ROCr then retries the whole 28.47 GiB. With THP `enabled=madvise`, `defrag=madvise`, the
2 MB faults in this `MADV_HUGEPAGE` region can do direct compaction and reclaim, which migrate movable pages, possibly
pages of this same range faulted in a moment earlier, and every move invalidates the walk. That fits the slow
standalone run (each page faulted ~4 times; huge pages were allocated for the whole range, yet with 22 GiB of it
resident only ~5 GiB of the machine's anonymous memory was in huge pages; compaction stalls) and the fast THP-off runs (one fault per page, no compaction), but I have not traced the
invalidations. It may be the same family as the reclaim / migration + KFD userptr stalls in `docs/AMD_HIP.md`
(#750, #1705, #2413). We also saw one `verify: timed out at layer 32 (#267)` on 0.1.44.1 the same day, but
I have not tested whether THP off changes those.

#### Workaround that works here

Start the server with THP disabled for it; the engine inherits it:

```c
// nothp.c - gcc -O2 -o nothp nothp.c ; usage: nothp <command> [args...]
#include <stdio.h>
#include <sys/prctl.h>
#include <unistd.h>
int main(int argc, char** argv) {
    if (argc < 2) { fprintf(stderr, "usage: nothp <command> [args...]\n"); return 2; }
    if (prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0) != 0) { perror("prctl(PR_SET_THP_DISABLE)"); return 1; }
    execvp(argv[1], argv + 1);
    perror("execvp");
    return 127;
}
```

`nothp python serve/server.py --engine strata --config <config.json>`. No root needed. Cost: about 5 s more for the
allocation in the fast state (8.2 s against 3 s); no measurable change in decode or prompt speed in the A/B above
(4 starts per arm, single requests, so only large differences would show).

#### Possible changes in Strata (suggestions; only the first is measured, and only as the process-wide workaround)

1. Disable THP only around the complement's allocation: `prctl(PR_SET_THP_DISABLE, 1)` before the `cudaHostAlloc`,
   restore the previous value after it. The faults happen inside that call. (Other threads' faults in that window
   lose THP too.)
2. Allocate the complement in 2 GiB pieces, as #2485 does for SYCL: 3.6 s in the fast state; not measured in the slow
   state, where a piece that is invalidated would retry only its own 2 GiB.
3. Print the allocation time separately from the copy in the `complement ready` line, so a slow start can be told
   apart from a slow read.

#### Not tested

System-wide `transparent_hugepage/defrag` = `defer+madvise` or `never`, or `enabled=never` (root); a kernel trace of
the invalidations (root); other ROCm or kernel versions; one GPU with the real engine; `HSA_USERPTR_FOR_PAGED_MEM=0`
(as far as I can tell it applies to paged allocations, not this one); what makes back-to-back starts slow on one day and fast
on another (our workload between starts is the obvious suspect).

<details><summary>pintest.cpp (standalone timing)</summary>

```cpp
// 2026-10-16: standalone timing of the page-locked host allocation Strata 0.1.44.1 makes for its RAM expert complement
// (FileExpertSource::pin_cache_complement: one cudaHostAlloc(bytes, Mapped|Portable) -> hipHostMalloc on ROCm), to
// show whether its 33-386 s spread is in the HIP/KFD allocation itself. No model, no kernels.
//
// Usage: pintest <mode> <GiB> [chunk_GiB] [repeats]
//   mode  alloc-mp   hipHostMalloc(Mapped|Portable)            - what Strata does
//         alloc-m    hipHostMalloc(Mapped)                     - one GPU's mapping only
//         alloc-d    hipHostMalloc(Default)
//         register   malloc + memset(0) + hipHostRegister(Mapped|Portable); the compiler may turn malloc + memset(0)
//                    into calloc (it did here: 0.00 s), so the pages are faulted inside hipHostRegister
//   chunk_GiB  0 = one allocation (default); N = ceil(GiB/N) allocations of N GiB
// Per repeat: the alloc time, then a one-thread memset of every byte (Strata copies the experts in next), then the free.
// /proc/vmstat counters (compaction, direct reclaim, swap, THP) are printed as deltas over the alloc.
#include <hip/hip_runtime.h>
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <map>
#include <sstream>
#include <string>
#include <vector>

static double now_s() {
    return std::chrono::duration<double>(std::chrono::steady_clock::now().time_since_epoch()).count();
}

static std::map<std::string, long long> vmstat() {
    static const char* keys[] = {"compact_stall", "compact_fail", "compact_success", "pgsteal_direct", "pgscan_direct",
                                 "allocstall_normal", "allocstall_movable", "pswpin", "pswpout", "thp_fault_alloc",
                                 "pgmajfault", "pgfault", "nr_free_pages", "thp_split_page", "thp_fault_fallback", nullptr};
    std::map<std::string, long long> m;
    std::ifstream f("/proc/vmstat");
    std::string k;
    long long v;
    while (f >> k >> v)
        for (const char** p = keys; *p; ++p)
            if (k == *p) m[k] = v;
    return m;
}

static std::string vmdelta(const std::map<std::string, long long>& a, const std::map<std::string, long long>& b) {
    std::ostringstream o;
    for (auto& [k, v] : b) {
        long long d = v - a.at(k);
        if (d != 0) o << " " << k << (d > 0 ? "+" : "") << d;
    }
    return o.str();
}

static long long meminfo_kib(const char* key) {
    std::ifstream f("/proc/meminfo");
    std::string k;
    long long v;
    std::string unit;
    while (f >> k >> v >> unit)
        if (k == std::string(key) + ":") return v;
    return -1;
}

#define CK(x)                                                                                          \
    do {                                                                                               \
        hipError_t e_ = (x);                                                                           \
        if (e_ != hipSuccess) {                                                                        \