#6554 [Engine] STRATA_NUMA_ARENA=1 on Windows: node-pinned first-touch slice placement for the expert arena
open · @noahark · 0 评论 · 去 GitHub 看
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
说明
## Summary `STRATA_NUMA_ARENA=1` places the expert arena's per-role slices on NUMA nodes 0 and 1 on **Windows**, the twin of the Linux placement (#6529; the two are OS-disjoint and share only the env name). On a dual-socket Windows host the default path draws the arena's physical pages from the registering thread's node and spills over the socket link when the node fills - the same pathology measured on Maya in mw00/project-maya#56 (82/18 ↔ 15/85 node flips, decode moving ~30% with the draw). ## What the change does `reserve()` skips the `MEM_LARGE_PAGES` attempt only when the placement can actually run - the switch set, a native layout it can place, and two NUMA nodes (a host or pack it cannot serve keeps its large pages) - and the first and `Deferred` constructors call `place_arena_numa_slices_win` before the register step: - node → `GROUP_AFFINITY` via `GetLogicalProcessorInformationEx(RelationNumaNode)`, filed by `NodeNumber`. `GetNumaNodeProcessorMaskEx` is not used: on Win10 19044 it writes a bare `GROUP_AFFINITY` per node (observed here), which reads back as garbage through the documented struct. - 8 toucher threads per node pinned with `SetThreadGroupAffinity` - `SetThreadAffinityMask` cannot address a second processor group (36+36 logical CPUs make two on the measurement host), which is what collapsed the pin masks in the #56-era "first-touch does not follow the touching thread" measurement. - the same page math as the Linux twin: within each expert's `[gate | up | down]` role, the page-aligned first half is touched on node 0 and the second half on node 1, plus the trailing spare, so the whole arena is resident before `cudaHostRegister` pins it. - the **ByNeed constructor is not placed** on purpose: pin-by-need pads every blob to a 4 KiB stride (`ArenaExpertSource`'s `PadMap`) and moves the destination offsets, so the packed-layout placement would misplace every blob after the first one. The other two constructors use the packed layout this function places. - a single-node host (or a failed topology query) logs `one NUMA node (or the topology query failed), slice placement not needed`; touchers that cannot be pinned turn the note into a `WARNING: slice placement partial`. Nothing changes for Linux, for single-node Windows, or with the switch unset; the placement happens once, at start. ## Why first-touch and not `VirtualAllocExNuma` The node argument of a commit only applies where the preferred node was recorded at `MEM_RESERVE` time - committing per-node ranges of an existing reservation cannot change it (measured in #56: 256 alternating `VirtualAllocExNuma` commits, 100% one node; re-confirmed here). Fresh per-node allocations do honor the node, but then the arena stops being one contiguous range, which the engine's single `cudaHostRegister` (and the whole registration/cap machinery) depends on. The working primitive is the documented first-touch preference through group affinity - a demand-zero page prefers the node of the CPU that faults it, falling back to another node when that one runs out - which the prototype (`512 MB`, `QueryWorkingSetEx` with the node field self-calibrated) places at run granularity **>= 128 KB**: 128/256/512 KB alternating blocks and contiguous halves all land ~50/50, below that the first 2 MB follow the touching thread and the rest collapses onto one node. The per-role halves here are ~256-307 KB, inside the working zone. Commit shape does not matter (one-shot commit, per-page `VirtualAllocExNuma`: identical distributions). `SV_PAGE_NUMA_AVAILABLE` (named in the #6023 comment) has no reachable value: no undocumented allocation bit is accepted on Win10 19044, and none is needed. ## Validation - **Environment**: 2x Xeon E5-2696 v3 (2 NUMA nodes), 128 GiB DDR3L-1600, Windows 10 Enterprise LTSC 21H2 (19044), Tesla V100-PCIE-32GB (TCC, CUDA 12.4, sm_70). Built from `6e38f58` (`v0.2.18`) + this patch, `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_BUILD_TYPE=Release`, `cmake -S engine -B build -G Ninja`. - **Form**: single GPU (`--gpu 2`), partial residency - the 56.2 GiB expert arena as Tier-1, 10120 experts in the GPU cache, `--ple-io direct`, `--spec 4 --mtp`, `--kv int8`, `--max-context 262144`. Full engine command line: `strata.exe --serve --pack D:\strata-data\packs\v31-ud-iq4-xs --native D:\models\strata\huihui-v2-v31-ud-iq4-xs\Qwen3.8-Flash-Next-UD-IQ4_XS-v31-00001-of-00004.gguf --expert-profile <profile.bin> --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.70 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 32768 --vram-reserve-mib 2000 --prefill-pipe 1536 --ple-io direct --gpu 2` (served through `serve/server.py --engine strata --config <json> --port 8095`; the two arms use the same binary and config and differ only in `STRATA_NUMA_ARENA=1`, each arm a fresh engine load). - **Placement** (`QueryWorkingSetEx` on the engine process, the node field self-calibrated per process; a uniform stride over the arena region capped at 65,536 samples per region - every ~227th page of the 56.2 GiB arena here, 65,525 samples): default arm **20.4% node 0 / 79.6% node 1** and 20.6/79.4 on a second fresh load (the draw, lopsided both times); with the switch **46.6/53.4** and **50.8/49.2**. The ~3-point shortfall from exactly 50/50 on the first start is most plausibly the fallback the first-touch rule itself allows (node 0's free memory was tight around the placement, though the per-node free levels were not sampled at that moment), and the placement's own log says how long it took: `large pages skipped (STRATA_NUMA_ARENA=1); using 4 KB pages; NUMA slice-placed (STRATA_NUMA_ARENA=1: slice 0 -> node 0, slice 1 -> node 1 in 14.90 s)`, with `cudaHostRegister PORTABLE ok` over the fully-faulted arena. - **Speed** - 3 rounds per arm of: a 23,410-token prompt, a 188,541-token prompt, then a ~380-token decode in the churned state the prompts leave behind. Protocol: the arms ran sequentially (default first, then the switch - not interleaved), one fresh engine load each, and the table's rounds are from that one load per arm. On this host the default arm also runs 4 KB pages (the large-page attempt is refused for lack of `SeLockMemoryPrivilege`), so page size is not a confound here: | round | arm | 23K-token prompt | 188K-token prompt | churned decode | |---|---|---:|---:|---:| | 1 | default | 1,564 tok/s | 1,301 tok/s | 20.4 tok/s | | 2 | default | 1,569 | 1,302 | 24.0 | | 3 | default | 1,535 | 1,296 | 23.4 | | 1 | `STRATA_NUMA_ARENA=1` | 1,572 | 1,305 | 37.3 | | 2 | `STRATA_NUMA_ARENA=1` | 1,553 | 1,305 | 39.6 | | 3 | `STRATA_NUMA_ARENA=1` | 1,555 | 1,302 | 31.1 | Prefill: medians within 0.6% (per-round spread up to ~1%). The churned-state decode improves **23.4 -> 37.3 tok/s median (+59%; per-round 20.4-24.0 vs 31.1-39.6, n=3, same binary/config, arms differing only in the env)**. That state reads swapped-back experts from the arena while decode runs, and the gain is larger than the PCIe path alone can explain: the #56 H2D gap (9.5 vs 11.0 GB/s, the card hangs off node 0 on this host - the `bw_dma` measurement from that issue) applied to a 20/80 -> 50/50 change bounds the link effect at ~4%. The rest is most plausibly CPU-side - in a partially-resident form the CPU pool computes non-cached experts, and under the default draw its arena reads land mostly on the far node's memory controller and the socket link. **We have not isolated this**; an all-on-node-0 arm and a CPU-pool expert count are the obvious follow-ups. - **Outputs**: the k-th round's completion text hashes identically across the two arms (first 16 hex of SHA-256 over the completion text: `aa333286852d70eb`, `8a848bb5308c504c`, `b97aa402a222c714`) - the placement moves pages, not bytes. Rounds differ from each other on both arms (the adaptive arena↔VRAM swaps change which experts the GPU cache computes vs the CPU pool, which the engine documents as a slight answer difference); the cross-arm, per-round identity is the bit-exactness claim. - **Steady-state decode** (hot GPU cache, arena rarely read): 72.3/76.3/76.8 tok/s (default, median 76.3) vs 62.0/72.5/77.3 (switch, median 72.5) - the medians differ by -5% (one switch round at 62.0), which is within the swap-state spread these repeats show; undecided, and the mechanism predicts parity here. ## Limitations (measured scope) - One host, one form: dual-socket Haswell, TCC V100, partial residency, `--spec 4 --mtp`, `--ple-io direct`. Not measured: WDDM hosts, multi-GPU layer splits, large-page-capable hosts (on those the switch costs 2 MB -> 4 KB pages when it runs), hosts with more than two NUMA nodes (the placement puts slices on 0 and 1 and says so in its note), the Windows HIP build (this code compiles under `_WIN32 && STRATA_USE_HIP` too), and the Windows NUMA_SPLIT pool form (off on Windows: "the split needs Linux"). - The ByNeed constructor is excluded by design (see above) and keeps its large pages; only the first and `Deferred` constructors place slices, and only the first was exercised by the A/B above. A single-node host was not available - the fallback path is code-reviewed, not machine-tested. - Startup cost: the switch pre-faults the whole arena through the pinned touchers (56.2 GiB in ~12-15 s here) before the register step - the opposite order of the default path's register-before-touch, whose comment warns an already-faulted register "is far more expensive and may fail outright"; on this host it registered `PORTABLE ok` every time, on one machine only. - The +59% decode mechanism is not isolated (see the speed section). ## Notes - Coordination with #6529: same env name, OS-disjoint code (independent `#if defined(__linux__)` / `#if defined(_WIN32)` blocks). Both PRs touch `reserve()` and insert before the register step in the same constructors, so the later-merged one will need to consolidate the call sites into a shared cross-platform entry (the slice page math is identical and wants to be a shared helper); happy to do that consolidation here if #6529 lands first. - The #56 dead-end list says "first-touch does not follow the touching thread" on Windows - that measurement's pin masks had collapsed into group 0 (a `ulong` marshal of a struct API plus `SetThreadAffinityMask` not addressing group 1); with group affinity the rule holds at the run granularities above, which is what this change relies on.
本站相关内容
相关页面的快捷入口。