#5949 [Engine] HIP/Linux: make HSA_USERPTR_FOR_PAGED_MEM=0 the default on discrete AMD cards
open · @alexanderi96 · 3 コメント · GitHub で見る
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
本文
### Area [Engine] ### What do you want to do? Make `HSA_USERPTR_FOR_PAGED_MEM=0` (and `HSA_SVM_FOR_PAGED_MEM=0`, which ROCm 10.2 and later also need) the default on Linux with discrete AMD cards, instead of a line in the troubleshooting docs. I'm opening this to agree the scope first, as CONTRIBUTING asks for a change to a default. Why: without it, the expert memory in system RAM is ordinary pageable memory that the GPU reads through userptr. When the kernel moves or reclaims those pages, amdgpu stops all of the engine's queues until they are back. Under memory pressure this is what ends in `no progress for 60 s` / `verify: timed out` (#649, #1705, #4862, #4866). Most people hit it before they find the docs. With the variable set the memory comes from GTT, which amdgpu pins, and the stops go away. Measured on one RX 7900 XTX, 64 GB, Swift 1.5 IQ3_S, budget 24 GiB, mean of three cold starts with five 44K-token prompts each (details in #3913): | | Another process touching 14 GB during the prompts | Start with the page cache full | | --- | --- | --- | | v0.2.13, default | 15 of 15 requests failed, 22 watchdog restarts | 134 s of queue stops, 1,578 tok/s | | v0.2.13, variable set | 0 failed, 0 s of stops | 134 s of stops (#1250 paced path), 2,244 tok/s | | v0.2.13 + #3913, variable set | 0 failed, 0 s of stops | 0 s of stops, 2,431 tok/s | Other reports with GTT-backed memory: tproos on #3913 (2 x RX 9070 XT, no more -15 exits) and vivienfanghuagood in #4866 (two sockets, NUMA balancing: 0 s of stops). ### What has to be true before it can be the default 1. **The GTT limit.** amdgpu allows half the RAM in GTT by default (`ttm.pages_limit`). A resident budget plus KV and buffers above that fails to allocate. The engine would need to fit the budget to the GTT limit by itself when the variable is on (reading `mem_info_gtt_total` / `mem_info_gtt_used` of the card) and say so in the log. I can do this as a separate PR. 2. **The start-up path.** With the page cache full, the #1250 paced path registers pageable memory again, so the variable alone doesn't help there. #3913 fixes this. 3. **Where to set it.** ROCr reads the variable when it starts, so either setup writes it in the config's `env`, or the engine sets it itself before the first HIP call. I'd lean towards the engine, so a plain start matches the app and the Terminal (as 0.2.13 did for other settings). Either way a value the user sets is kept, and `HSA_USERPTR_FOR_PAGED_MEM=1` turns it off. 4. **Who it applies to.** Linux, HIP, discrete cards only. Integrated graphics (Strix Halo, 780M) are left out. CUDA, SYCL, Metal and Windows are unchanged. ### Costs to accept - GTT pages can't be swapped: what the engine holds is taken from the rest of the host, as with mlock. - Pages kept by TTM's pool after the engine stops can halve the budget on a quick restart unless `ttm.page_pool_size` is set (in `docs/AMD_HIP.md`). - #2729 (RX 7900 XT, 31 GiB RAM) still saw SVM restore hangs with the variable set. It should be checked again with `HSA_SVM_FOR_PAGED_MEM=0` on a small-RAM host before the default changes. ### What do you do today instead? Set both variables in the server JSON's `env` and raise `ttm.pages_limit` by hand. Is this a direction you'd accept? If yes, I'd go in this order: #3913, then the GTT fit, then the default.
関連リンク
インストール・モデル・リリースへのサイト内リンク。