Pull requests / #2242
#2242 numa-placement: --numa-hot-node N - keep the GPU-side host pools on the GPU's NUMA node (Linux, dual-socket, opt-in)
open · @G0-0000 · 0 comentarios · En GitHub
Setup & installMulti-GPUNVIDIA / CUDADocumentationWindowsLinux
Descripción
## What it changes New opt-in engine flag `--numa-hot-node <n|off>` (Linux, dual-socket hosts): - **The resident complement** (the page-locked RAM copy of the experts the GPU cache does not hold) is claimed on node N **before** the expert cache's startup fill reads tens of GiB through the page cache (mmap -> mbind(MPOL_BIND) -> pre-touch -> the existing register+lock path consumes it). - **The five pinned host pools** the per-token path touches - the K/V streaming host copy, the doorbell, the host embedding, the session loop scratch and the weight staging buffers - allocate with the calling thread bound to node N. - **A conversation cache's parked segments** (16 MiB glibc segments, bulk snapshot data) are faulted in on the **cold** node - derived as the first node that is not the hot one - leaving the GPU's node to the pools above. - The NUMA primitives (mbind/set_mempolicy wrappers) moved from expert_source.cpp's anonymous namespace into a shared `numa_policy.hpp`, and `docs/AI_SETUP.md` / `docs/DETAILS.md` document how to find the GPU's node at install time. ## What it leaves alone With the flag absent or `off`, **no syscall is issued** and every allocation, output and log line is byte-identical to before. `0` is a valid node number (bind to node 0); only `off` - or the flag's absence - turns the feature off. A single-node host has no cold node and the S3 part does nothing there. ## Why #1900 sketched this. On a two-socket host the GPU hangs off one node; the engine's hot host pools want to be on that node, but the startup fill's page cache and the complement used to race for it, and under memory pressure the complement ended up on the other socket (measured 80.6% before, 100% after, stable across restarts). ## How it was measured Supermicro X10DRC (dual Xeon Broadwell-EP), Tesla V100-SXM2-32GB, Qwen3.8-Flash-Next UD-IQ4_XS: - complement: 31.42 GiB, 100% on the GPU's node (`bind:1`), claimed in 18-37 s before the fill; - the five pinned pools: 100% on the GPU's node (unchanged steady state - the win is determinism under pressure); - parked conversation snapshots: 89-100% on the cold node; - decode/prompt speeds unchanged (the placement moves pages, not the token path), verified with interleaved A/B. ## How it was tested GPU-free: complement-claim unit tests (per-page node check via get_mempolicy, duplicate-claim and out-of-range rejection) and the full non-CUDA ctest subset. On-GPU: startup runs, parity runs, and three controlled stop-the-service windows comparing numa_maps aggregates. ## Open question The node is read once at install time (`cat /sys/bus/pci/devices/<GPU-BDF>/numa_node`) and written into the configuration - we did **not** add runtime auto-detection, on purpose. If you would rather have setup's check print a suggestion (or anything else around this), we are happy to do it.
En el sitio
Enlaces a install, modelos, releases.