#5436 [Engine] opt-in STRATA_ARENA_NUMA: place the expert arena per NUMA node on two-socket Linux
open · @Keoian · 0 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
Beschreibung
**Problem and result.** On a two-socket Linux machine the CPU pool splits itself by NUMA node (0.2.2), but when the experts come from the arena (the default source whenever RAM holds every expert) their memory is not placed per node: the engine logs `NUMA split on, but this model's experts come from the arena path: memory is not placed per node (known limitation)`. `cudaHostRegister` faults the whole arena in from one thread, so it lands on that thread's node and the other node's slice of the pool reads every row across the socket link. Measured on 2x Xeon E5-2696 v3 (DDR3-1600, one DIMM per channel) with 2x Tesla P100 16 GB, Unsloth-layout Qwen3.8-Flash-Next (bitlamas Q4_K_XL-IQ4NL, 63.5 GiB of experts): the CPU pool took 28.6 ms per decode window for ~351 expert-blob reads (~34 GB/s; one socket alone streams 45.9 GB/s). This PR adds an opt-in `STRATA_ARENA_NUMA=1` (Linux) that places each expert's row halves (`numa_slice_ranges`, the pool's slices) on their slice's node before the registration. With it the CPU pool takes 14.0 ms per window (same config, one run each), the output is unchanged (teacher-forced log-probabilities byte-identical), and the placement takes ~2 s at start. Default: nothing changes except one log line (the known-limitation line names the switch). **Related discussion.** #3708 (the arena-path limitation on three two-socket boxes). There petski3 described and measured the same direction on gfx906 (a hook before `cudaHostRegister`, 4 KiB pages, node-pinned touchers, `move_pages` checks; +29% decode on their box) and offered a PR "if this direction is wanted". This branch arrived at the same design independently (both follow the file path's `NumaComplementPlacement`; I found their comment only when searching before this PR), on CUDA, and adds what a fourth box needed: a long-term pin of each chunk before it is touched (without it 21-32% of one node's half migrated away during the registration) and a PLE table read half per node (without it a cold start could leave one node too little room for its half). #3730 keeps the automatic pool off on the arena path when memory is not placed; with this switch the placement can be checked and the split kept. Happy to fold this into petski3's PR instead if they open one. **Approach.** - A `pretouch` hook on `PinnedArena`'s bounds constructor (default empty) runs after the reservation and before the registration. `ArenaExpertSource::open` sets it when `STRATA_ARENA_NUMA=1` and the arena is a private, whole-registered arena of a native pack (otherwise it logs why not). - The touch: `MADV_NOHUGEPAGE` first (a 2 MiB THP page holds about four row halves of both slices; with THP the halves landed ~50/50, measured), then 8 threads per node, each bound with `set_mempolicy(MPOL_BIND)` to its node (the node's file cache is reclaimed instead of spilling; `MPOL_PREFERRED` when the node cannot hold its slice), write one byte per page of their node's ranges. - **Long-term pins from the touching threads, before the touch.** After the touch was 100% local, sampling after the registration still found 21-32% of node 1's half on node 0 (863-1313 of 4096 sampled pages, three starts). Reproduced outside the engine: at 63.5 GiB, `cudaHostRegister` alone migrated 1.14 M pages. Node 1 has 7329 CMA-type pageblocks (14 GiB; `CmaTotal: 0`); an anonymous page there cannot stay under a long-term pin, and the migration takes a free page of the other node before it frees file cache on its own. So each touching thread takes an io_uring fixed-buffer registration (a FOLL_LONGTERM pin) of each chunk and releases it at once. Revision 2 takes it **before** the chunk is touched: the pin's own faults then allocate the pages outside CMA-type memory under the thread's `MPOL_BIND` (the node's file cache is reclaimed), and nothing migrates. (The first revision pinned right after the touch: the pages migrated while the node had room, which held in most starts but left 96.8% of a half local in a cold start of the stock config and 81.7% in one of the author's tuned config, see Validation.) `STRATA_ARENA_NUMA_PREPIN=after` is the first revision's order, `=0` no pre-pin. `mlock2(MLOCK_ONFAULT)` keeps reclaim off the fresh pages until the registration; one `mbind(MPOL_BIND, both nodes)` keeps NUMA balancing off the VMA. Both need `RLIMIT_MEMLOCK` (`ulimit -l unlimited`, systemd `LimitMEMLOCK=infinity`); without it the placement runs unprotected and the log says so. - **The room check** (`MPOL_BIND` only when the node can hold its slice, else `MPOL_PREFERRED`) leaves out the node's CMA-type memory (`/proc/zoneinfo` `cma`) and its locked pages: a long-term pin can use neither, and under `MPOL_BIND` a pin with no room has nowhere else to go. The start line prints it (`room 45.0 / 42.7 GiB`). - **A PLE table locked in RAM is read half per node** (`src/kernels/ngram.cpp`, only with `STRATA_ARENA_NUMA=1` and only when the table is mmap-locked). The table is locked before the arena is filled; read from a cold file after one `MADV_WILLNEED` from the main thread, all of it (26.8 GiB here, 7.0 M pages) landed in that thread's node, which then had 31.4 GiB of room for a 31.9 GiB half. Now 8 threads per node fault in one half each (`MPOL_PREFERRED`, `MADV_WILLNEED` per 64 MiB piece), and pages an earlier reader left in the file cache on the wrong node are moved (`move_pages`; 3.5 M pages in 1.7 s on a warm start). The line `PLE table per NUMA node ...` reports it. `STRATA_PLE_NUMA=0` keeps the plain fault-in. - `move_pages` samples each slice's pages before the registration and after the load; the line `STRATA_ARENA_NUMA: row halves placed before the registration: ...` reports both. After the pool is built, generate.cpp checks the placement's nodes against the pool's slices. - Linux only; the io_uring and `mlock2` parts compile out when the headers lack them. Nothing changes where a page's contents come from or what is computed: only which node a page is on. **Validation** (Ubuntu 26.04, Linux 7.0.0-43, CUDA 12.8.93, gcc-14, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60`, 2x E5-2696 v3, 2x Tesla P100 PCIe 16 GB, driver 580.178.04) Passed, revision 2 (this branch's head): - CUDA build of `strata`. - Placement from a cold file cache (`sync; echo 3 > /proc/sys/vm/drop_caches` before every start, as after a boot), the stock setup config with `STRATA_ARENA_NUMA=1`, revision 1 (R, commit 2850a55f) and revision 2 (N) interleaved RNNR, a fresh server per block. Pages of each slice on its node after the registration and the load, of 4096 sampled: | block | slice 0 | slice 1 | placement time | load (cold) | |---|---|---|---|---| | R1 | **3963** | 4096 | 10.8 s | 260 s | | N1 | 4096 | 4096 | 1.7 s | 244 s | | N2 | 4096 | 4096 | 1.7 s | 244 s | | R2 | 4096 | 4094 | 7.6 s | 252 s | On the author's tuned config (the same engine change there) revision 1's order also gave 3346/4096 and 4075/4096 and, after a real cold boot, 88.3%; revision 2 4096/4096 on both nodes in every start (two cold, one warm). - Decode is unchanged by revision 2 (the hot path is the same): time per decode step N/R, decode_ms / (generated - accepted), code 1.004, prose 1.009, L8k 0.982, geomean 0.998 (medians in tok/s N / R: 39.82 / 40.23, 30.67 / 31.77, 33.38 / 33.69; they move with each run's draft acceptance). - Output: `tools/quant_quality.py run` (as below, CPU experts, FP16 K/V) with revision 2 and `STRATA_ARENA_NUMA=1`: the LOGPOS file is byte-identical to the run without the switch (md5 `989d6f4fd04641f8bfa4bb96f9446b0e`); 25.3-29.4 s per 1000-token segment (revision 1: 25.2-26.0 s). Passed, revision 1 (the measurements below are of the first revision; revision 2 changes only the start): - CUDA build of `strata`. - Output: `tools/quant_quality.py run` (5 x 1000 tokens of `data/quant_quality/v1`, CPU experts, FP16 K/V) with and without `STRATA_ARENA_NUMA=1`: the two LOGPOS files are byte-identical (md5 `989d6f4fd04641f8bfa4bb96f9446b0e`). The same with GPU experts (`--gpu-experts auto`) together with `STRATA_DOWN_WS=1 STRATA_SPEC_COST=1`: byte-identical to the run without them. - Placement (`move_pages`, 4096 samples per slice) after the registration and the load: slice 0 4096/4096 on node 0, slice 1 3896/4096 on node 1 (without the pre-pin: 2783-3233/4096). - Speed, the CPU path alone (the yardstick above, every expert on the CPU): 39.4 / 39.1 / 37.6 / 38.8 / 39.5 s per 1000-token segment without, 26.0 / 25.9 / 25.2 / 25.9 / 25.8 s with. - Speed, 2-GPU layer split decode (`STRATA_DECODE_TIMING=1`, one code prompt, 512 tokens): CPU pool 28.6 -> 14.0 ms per window (the stock config, one run each). End to end, the stock setup config with and without the switch, ANNAANNA, a fresh server per block: decode tok/s, median [min-max]; prompt reading unchanged (L8k, 15.4K tokens: 308.3 vs 308.2 tok/s). The GPUs are the bottleneck of this config, so the CPU's gain shows only in part: | case | without (4 runs) | with `STRATA_ARENA_NUMA=1` (4 runs) | median gain | with > without | |---|---|---|---|---| | code | 38.42 [36.66-38.66] | 40.25 [40.09-40.86] | +4.8% | 4/4 | | prose | 30.18 [29.65-30.62] | 31.60 [31.18-31.92] | +4.7% | 4/4 | | chat | 27.62 [23.48-28.29] | 29.93 [29.62-32.16] | +8.4% | 4/4 | | L8k | 31.48 [31.30-31.54] | 34.35 [33.58-34.93] | +9.1% | 4/4 | - Without `RLIMIT_MEMLOCK` the engine starts, places, and logs `NOT locked (RLIMIT_MEMLOCK)` / `long-term pre-pin unavailable`. Failed: none observed in the final form. Iterations that did not hold the placement: THP left on, an `mbind` of the VMA alone, `mlock2` alone (above); the PLE table halved with the touch-then-pin order (slice 1 3559/4096, a warm start); the pin before the touch without halving the PLE table (slice 0 3240/4096 from a cold cache: the room check found 31.4 GiB on node 0 for 31.9 GiB and chose `MPOL_PREFERRED`). Not run: - HIP and SYCL builds (`expert_source.cpp`, `pinned.cu`, `ngram.cpp` are in them): no ROCm or oneAPI here. Hence a draft. - Windows (the code is under `__linux__`). - Another two-socket machine, a machine without CMA-type pageblocks, more than two NUMA nodes (the switch then logs "off"). - The canonical (Q2_0 `experts.bin`) arena: only native packs are placed, as the pool only splits native layers. - A PLE table read through the direct reader (`--ple-io direct`) or not locked: not placed (only the mmap-locked table). **Limits.** Exactly two NUMA nodes with CPUs; a private arena registered whole (not pin by need, not a shared arena, not the deferred Windows registration). The placement costs ~2 s per start at 63.5 GiB (1.7-1.8 s; revision 1's touch-then-pin took 7.6-10.8 s). A layer-affine variant (whole layers on each GPU's socket, so every RAM-to-GPU copy is socket-local) was measured and is not included: with `iommu=pt` the copies were equally fast either way, and decode lost 11-28% to the pool's remote reads. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01N976PznRue6hXdFtmrbJNj
Mehr auf der Site
Links zu Install, Modellen, Releases.