#5949 [Engine] HIP/Linux: make HSA_USERPTR_FOR_PAGED_MEM=0 the default on discrete AMD cards

open · @alexanderi96 · 3 commentaires · Sur GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

### Area
[Engine]

### What do you want to do?
Make `HSA_USERPTR_FOR_PAGED_MEM=0` (and `HSA_SVM_FOR_PAGED_MEM=0`, which ROCm 10.2 and later also need) the default
on Linux with discrete AMD cards, instead of a line in the troubleshooting docs. I'm opening this to agree the scope
first, as CONTRIBUTING asks for a change to a default.

Why: without it, the expert memory in system RAM is ordinary pageable memory that the GPU reads through userptr.
When the kernel moves or reclaims those pages, amdgpu stops all of the engine's queues until they are back. Under
memory pressure this is what ends in `no progress for 60 s` / `verify: timed out` (#649, #1705, #4862, #4866). Most
people hit it before they find the docs. With the variable set the memory comes from GTT, which amdgpu pins, and the
stops go away.

Measured on one RX 7900 XTX, 64 GB, Swift 1.5 IQ3_S, budget 24 GiB, mean of three cold starts with five 44K-token
prompts each (details in #3913):

| | Another process touching 14 GB during the prompts | Start with the page cache full |
| --- | --- | --- |
| v0.2.13, default | 15 of 15 requests failed, 22 watchdog restarts | 134 s of queue stops, 1,578 tok/s |
| v0.2.13, variable set | 0 failed, 0 s of stops | 134 s of stops (#1250 paced path), 2,244 tok/s |
| v0.2.13 + #3913, variable set | 0 failed, 0 s of stops | 0 s of stops, 2,431 tok/s |

Other reports with GTT-backed memory: tproos on #3913 (2 x RX 9070 XT, no more -15 exits) and vivienfanghuagood in
#4866 (two sockets, NUMA balancing: 0 s of stops).

### What has to be true before it can be the default
1. **The GTT limit.** amdgpu allows half the RAM in GTT by default (`ttm.pages_limit`). A resident budget plus KV and
   buffers above that fails to allocate. The engine would need to fit the budget to the GTT limit by itself when the
   variable is on (reading `mem_info_gtt_total` / `mem_info_gtt_used` of the card) and say so in the log. I can do
   this as a separate PR.
2. **The start-up path.** With the page cache full, the #1250 paced path registers pageable memory again, so the
   variable alone doesn't help there. #3913 fixes this.
3. **Where to set it.** ROCr reads the variable when it starts, so either setup writes it in the config's `env`, or
   the engine sets it itself before the first HIP call. I'd lean towards the engine, so a plain start matches the app
   and the Terminal (as 0.2.13 did for other settings). Either way a value the user sets is kept, and
   `HSA_USERPTR_FOR_PAGED_MEM=1` turns it off.
4. **Who it applies to.** Linux, HIP, discrete cards only. Integrated graphics (Strix Halo, 780M) are left out.
   CUDA, SYCL, Metal and Windows are unchanged.

### Costs to accept
- GTT pages can't be swapped: what the engine holds is taken from the rest of the host, as with mlock.
- Pages kept by TTM's pool after the engine stops can halve the budget on a quick restart unless
  `ttm.page_pool_size` is set (in `docs/AMD_HIP.md`).
- #2729 (RX 7900 XT, 31 GiB RAM) still saw SVM restore hangs with the variable set. It should be checked again with
  `HSA_SVM_FOR_PAGED_MEM=0` on a small-RAM host before the default changes.

### What do you do today instead?
Set both variables in the server JSON's `env` and raise `ttm.pages_limit` by hand.

Is this a direction you'd accept? If yes, I'd go in this order: #3913, then the GTT fit, then the default.

Sur le site

Liens install, modèles, releases.