#4867 [Engine] HIP/Linux: keep the kernel's NUMA balancing off the engine's memory (GPU queue evictions)

closed · draft · @vivienfanghuagood · 0 comentários · No GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

## Problem and result

On a Linux host with `kernel.numa_balancing=1` (the default on most two-socket machines), the HIP engine stops at
random points of a request, in the prompt read and in decode, until the #29 watchdog ends it:
`no progress for 60 s during a request (...)`.

The cause is the NUMA-balancing scanner. Its PTE updates (`PROT_NONE` sampling) are MMU-notifier invalidations of the
`hipHostRegister`ed, KFD userptr ranges the GPU reads. On a GPU without XNACK, amdgpu answers each invalidation by
evicting all of the process's GPU queues until the range is validated again. Details, the ruled-out alternatives and a
standalone reproducer are in https://github.com/Niko1221/Strata/issues/4866.

This change sets `MPOL_LOCAL` first thing in `main()` when balancing is on and nobody chose a policy. MPOL_LOCAL is an
explicit policy with the same placement as the default, and the scanner skips memory under an explicit policy.

Radeon PRO W7900D (gfx1100), two-socket host, Ubuntu 24.04 / kernel 6.8, ROCm 7.2.1, c62d5e3a1. Qwen3.8-Flash-Next
IQ3_S with setup's config. Each row is 20 fresh 1K-token prompts with 64 output tokens:

| Arm | Watchdog stops | KFD `evicted_ms` per engine start | Decode (median) |
| --- | ---: | ---: | ---: |
| before | 7 / 20 | 77,147 to 236,059 | 62 tok/s |
| **after (this change)** | **0 / 20** | **up to 907** | **62.4 tok/s** (TPOT 16.0 ms, TTFT 1.53 s) |

The host's kernel log tells the same story:
- Before: 9 hung-task reports in 3 hours while the unpatched engine ran, all in `Workqueue: events_freezable
  amdgpu_amdkfd_restore_userptr_worker`, blocked 122 to 245 s.
- After: none in over 2 hours and more than 20 engine starts with this change or a `numactl` policy.

## Related discussion

https://github.com/Niko1221/Strata/issues/4866. Since this changes a default (on Linux when `numa_balancing` is on), it can be made opt-in if you prefer.

Same mechanism as #649 and #3913 (non-XNACK queue eviction on userptr invalidation), different trigger: no swap or
reclaim here, plenty of free RAM. #3913 locks the registered ranges against reclaim; we did not test whether
`mlock` also keeps the NUMA scanner away. Possibly related: #4363, #3443, #1705 (untested here).

## Approach

The new `include/strata/platform/numa_balancing.hpp` is called at the top of `main()` in `strata` (the server's
`--serve` engine) and `strata-dsv41`. On Linux, when `/proc/sys/kernel/numa_balancing` is non-zero and
`get_mempolicy` reports the default policy, it calls `set_mempolicy(MPOL_LOCAL)` and logs one line.

- **Why the scanner then skips the memory:** `task_numa_work()` passes over a VMA when `vma_policy_mof()` is false,
  which is the case for an explicit policy without `MPOL_F_NUMA_BALANCING`. Placement is the same as the default's, so
  the NUMA pool's first-touch placement is unchanged, and so are the output tokens.
- **It must run before any thread exists.** `set_mempolicy` is per thread, and the scanner judges a VMA by the policy
  of the thread it interrupts. The helper counts `/proc/self/task` and warns when other threads already exist. On
  this build `main()` starts with one. In the standalone reproducer, setting the policy after the CPU threads started
  left stalls of 44 to 46 s; setting it before left none over 100 ms in one round.
- **What is kept:** a policy the user set (`numactl --membind`, `--interleave`, ...), and the DeepSeek path's
  per-slice `mbind`.
- **Opt-out:** `STRATA_NUMA_BALANCING=keep`.
- **Why process-wide:** it covers every `hipHostRegister` call site (arena, KV host copy, file tier, DeepSeek) and
  future ones. It also avoids splitting the arena into many VMAs (the `vm.max_map_count` concern in
  `expert_source.cpp`).

An alternative needs no code change: operators can set `kernel.numa_balancing=0` on the host, or start the engine
under `numactl --localalloc`.

## Validation

**Passed:**
- `numa_balancing_test` (new, header-only): the decision table, built with g++ 13 and with the HIP build.
- HIP build for gfx1100 with setup's flags (`STRATA_PREFILL_MMQ`, `STRATA_Q6K_EXPERTS`, `STRATA_MMQ_KQUANTS`) at
  c62d5e3a1. `strata --version` on a balancing host prints the new line and no thread warning.
- Served run through `serve/server.py`, card used alone: 20 / 20 fresh 1K requests, no watchdog stop, the engine log
  shows the new line, `evicted_ms` up to 907 for the whole run (table above). The same 20 requests on the unpatched
  build in the same session: 7 stops.
- Wrapper equivalents of the policy, 20 requests each, no stop: `MPOL_LOCAL` (up to 3,223 ms evicted),
  `MPOL_INTERLEAVE` 0,1 (0 ms).
- All our runs with this change, the longer benchmark included (1K / 8K / 32K prompts, 1 / 4 / 16 concurrent
  requests): 25 engine starts, 260 requests served, 0 watchdog stops, and no hung task in the kernel log.

**Failed:**
- None observed.

**Not run:**
- CUDA, SYCL and Metal builds. The change is plain C++ behind `__linux__` and needs no GPU API, but it touches
  `generate.cpp` and `ds4_main.cpp`, which those builds compile, so this stays a draft until they are built.
- Windows: there the header compiles to nothing; not built.
- `strata-dsv41` at runtime: it is built, not run with this change.
- Single-socket hosts, where `numa_balancing` is usually 0 and the change does nothing; not tried.
- Multi-GPU, batch slots, the picture reader (a separate process, not patched).

## Limits

- One machine, one GPU model, one kernel version (6.8).
- Small residual evictions remain, up to 0.9 s per run in total, likely from other invalidation sources such as
  compaction. They are far below the watchdog.
- With `STRATA_NUMA_BALANCING=keep` (or a user policy that keeps `MPOL_F_NUMA_BALANCING`), the stalls come back.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.