#3413 [Engine] NUMA-split CPU expert pool, on by default on dual-socket Linux boxes

Descripción

## Problem and result

On a dual-socket box the CPU expert pool either stays on one socket (the 0.2.0 default, 11 workers here) or spreads across both and gets slower, as the benchmarks behind the 0.2.0 default (#674, #707) found. With the split, both sockets' cores pay off. As agreed in #3201, it is on by default when the pool's CPUs span 2 NUMA nodes, and `STRATA_NUMA_POOL=0` turns it off.

Box: 2x Xeon Gold 6226, 64 GB (32 GB per NUMA node), RTX 3090 on PCIe 3.0, Qwen3.8-Flash-Next-Uncensored IQ4_XS at 262K context, kernel booted with `iommu=pt`. Server decode:

| Arm | Workers | Median tok/s | Range |
|---|---|---|---|
| 0.2.0 default | 11, one socket | 54.05 | 50.7-54.8 |
| no split | 22 | 43.25 | 41.0-47.4 |
| split | 12 (6+6) | 51.45 | 50.0-52.9 |
| **split** | **22** | **59.3** | **57.1-61.9** |

+9.7 % over the default. Two earlier sessions, with the IOMMU translating DMA (the Debian default), gave +14 % and +11.5 %: the default gained from the faster bus, the split did not move (details below). A split without the placement (diagnostic build, earlier session) gave 51.0 against 53.1 for the default. Neither more cores, nor local memory at 12 workers, nor the per-node pool slices without the placement explains the gain: it takes both halves at 22 workers. Prompt speed and VRAM hit rate are unchanged. A machine with one NUMA node runs exactly as before: same worker count, no log line. A test checks it; no single-socket machine was at hand, so it ran on this box with the process bound to one node (`numactl --cpunodebind=0`), which the pool's topology detection follows.

## Related discussion

https://github.com/Niko1221/Strata/issues/3201 (scope agreed: on by default with two NUMA nodes, `STRATA_NUMA_POOL=0` to turn it off). #2242 took a different route, binding the complement to the GPU's node, and was declined for lack of dual-socket measurements.

## Approach

- **Pool:** one task slice per NUMA node in the native gate/up and down phases, each with its own claim counter and barrier, so a worker claims and waits only within its own node's slice.
- **Placement:** the rows of every complement expert in a native layer are cut in two (gate/up on n_ff, down on n_embd, so no partial-sum reduce), and each half is first-touched on its node by threads pinned there, after the page cache of the model files, the pack's files and the MTP head is released. Releasing only the GGUF maps left the pack and MTP pages read at start in the cache, and a 32 GB node fell under 95 % of its slices. Placement is checked after registration with move_pages samples and reported per node.
- **Workers:** under the split the default covers both sockets' physical cores, minus the host core and the usual spare core, capped at 32 (22 here); an explicit `--pool-workers N` alternates sockets. The default spans both sockets only when the split will follow (Linux, exactly 2 nodes, pinned threads, no `STRATA_EXCHANGE_ROTATE=1`), since 22 workers without the split are the slowest arm above.
- **Refusals:** a node count other than 2, `STRATA_EXCHANGE_ROTATE=1`, unpinned threads and a lopsided worker layout turn the split off, with the reason in the log; the pool then runs as in 0.2.0.
- **Test mode:** `STRATA_NUMA_POOL=test` builds the same slices on any machine, without placement.
- **Docs:** docs/DETAILS.md, next to `--aux-cpus`.

## Validation

**Passed:**
- Rebased on 0.2.1, CUDA 13.2 build on the box: `pool_plan_test` OK; `file_expert_source_test` PASS; `pool_tasks_test` "360 bitwise comparisons passed, 192 of them in NUMA-split mode", and its default check: unset 22 workers split on, `1` 22 workers split on, `0` 11 workers.
- Single-node machine, simulated: `pool_tasks_test` under `numactl --cpunodebind=0` on the same box: unset 11 workers split off, `1` 11 split off, `0` 11; the unset runs log nothing, `1` logs "NUMA split off: the pool's CPUs span 1 NUMA node(s)". Not run on a real single-socket machine.
- Server start on 0.2.1 with this branch, same box and model: unset gives `NUMA split on: slice 0 = node 0 (12 threads), slice 1 = node 1 (11 threads)`, 22 workers, placement 100.0 / 99.6 %; `STRATA_NUMA_POOL=0` gives 11 workers on socket 0 and no split line. One start per arm, a functional check, not a speed comparison.
- Opt-in version on v0.2.0-beta, CUDA 13.2 build on the box (Linux, RTX 3090, sm_86) at ca1dafbd: `pool_plan_test` OK; `pool_tasks_test` "360 bitwise comparisons passed, 192 of them in NUMA-split mode", 20 runs out of 20; `file_expert_source_test` PASS.
- Host-only g++ build (CUDA, HIP and SYCL off) on a single-socket laptop: no warnings, `pool_plan_test` and `pool_tasks_test` pass.
- Server decode, protocol below (opt-in version on v0.2.0-beta, same pool and placement code): default 54.05 tok/s, split 59.3, 22 workers without split 43.25, split at 12 workers 51.45. Placement 95.8 to 100 % per node on every split start (95.2 to 100 % in the earlier sessions).

**Failed:** none observed.

**Not run:**
- HIP, SYCL, Metal and MSVC builds: no toolchains here. The Linux-only calls (move_pages, sched_setaffinity, madvise, /sys reads) sit behind the guards the files already use; on other platforms the split stays off (`STRATA_NUMA_POOL=1` logs "NUMA split off: the split needs Linux").
- The split in pool modes 0 to 4: the test exists but needs AVX-512 VBMI, which the 6226 lacks, so it skips here.
- Other dual-socket hardware, more than 2 NUMA nodes, AMD NPS modes. A single-socket CPU exposing 2 nodes (SNC2, NPS2) would get the split by default, unmeasured.
- The split against the default with upstream's default tail skip (7) and pool spin (20 ms): every arm here ran with the tuning in the notes below.
- End-to-end output identity: greedy CLI outputs already differ between runs with the split off, because the CPU/GPU expert routing depends on timing. Correctness rests on the bitwise pool tests.

## Limits

- Linux, exactly 2 NUMA nodes.
- Only native expert layers are split; ArenaExpertSource and the cudaHostAlloc path are not (the engine logs it for the arena path).
- Each node needs room for its half plus 2 GiB. On this box node 0 is tight: the "NUMA placement at risk" warning fires on every start. Before the pack and MTP drop a node landed at 95.2 to 95.8 % on some starts, just above the 95 % FAILED line; with it, three starts placed 100 % on both nodes.
- On by default from one box's measurements. Tworks25 plans to report on a 2x E5-2690 v2 box (#3201).

<details><summary>Test protocol</summary>

Box: HP Z6 G4, 2x Xeon Gold 6226 (12 cores each, 2 NUMA nodes), 64 GB DDR4 (32 GB per NUMA node) on 4 of 6 channels per socket, RTX 3090 on PCIe 3.0 x16, CUDA 13.2, Debian 12 kernel 6.1 booted with `iommu=pt`. Engine at ca1dafbd on v0.2.0-beta, server config: `--pack orca-iq4_xs --native Qwen3.8-Flash-Next-Uncensored-IQ4_XS --expert-cache auto --resident-budget-gib 55 --prefill auto --spec 4 --spec-min-p 0.5 --mtp mtp/rt --mtp-max-t 2 --max-context 262144 --kv int8 --vram-reserve-mib 844 --adapt-async 1`, expert profile frozen (md5 checked before every start, profile saving off). Every arm ran with the same config env: `STRATA_POOL_SPIN_US=200`, `STRATA_ROUTE_TAIL_SKIP=0`, `STRATA_DECODE_TIMING=1`. Harness, configs and raw logs: https://gist.github.com/baptisterajaut/0cd495f3e548be4964406f1198119e28

Arms: measured before the switch became `STRATA_NUMA_POOL` and on by default. D = default (split off, 11 workers); S = split on (22 workers); X = split with `--pool-workers 12`; N = split off with `--pool-workers 22`. Server starts in the order D1 S1 D2 S2 X1 N1 S3 X2 N2 S4, no drop_caches between them (production-like). Per start: a 60K-token passkey prompt twice (prompt speed, passkey checked), then 3 decode samples of a fixed prompt (Python LRU cache with type hints plus 5 design bullets, max_tokens 3000, reasoning_effort low, sampled), decode speed read from the server's `timings.predicted_per_second`, VRAM hit rate from the server's done line. A split start below 95 % placement on a node would have been discarded; none was.

| Arm | Workers | Start 1 | Start 2 | Start 3 | Start 4 | Median (range) |
|---|---|---|---|---|---|---|
| D: default | 11, one socket | 54.0 54.1 54.7 | 50.7 54.8 53.8 | | | 54.05 (50.7-54.8) |
| N: no split | 22 | 43.5 42.5 47.4 | 43.0 41.0 45.2 | | | 43.25 (41.0-47.4) |
| X: split | 12 (6+6) | 50.8 50.0 50.8 | 52.1 52.9 52.7 | | | 51.45 (50.0-52.9) |
| S: split | 22 | 57.1 58.2 59.7 | 58.2 60.1 59.9 | 61.2 58.2 61.9 | 58.4 60.6 58.9 | 59.3 (57.1-61.9) |

Prompt speed 1282 to 1341 tok/s across all arms, passkey found in all 20 prompt runs, VRAM hit rate 72 to 79 %. N streams 4.0 to 4.7 % of the routed experts over PCIe against 1.2 to 2.4 % in the other arms.

Earlier sessions, same protocol and binary, IOMMU in translated mode (`DMA-FQ`, pinned H2D 7.7 to 9.2 GB/s against 12.3 with `iommu=pt`). First, order D1 S1 X1 D2 S2 X2 N1 S3 N2 S4:

| Arm | Workers | Start 1 | Start 2 | Start 3 | Start 4 | Median (range) |
|---|---|---|---|---|---|---|
| D: default | 11, one socket | 51.0 53.3 53.3 | 51.0 53.1 52.1 | | | 52.6 (51.0-53.3) |
| N: no split | 22 | 40.5 50.4 50.1 | 45.3 50.7 53.2 | | | 50.25 (40.5-53.2) |
| X: split | 12 (6+6) | 50.4 52.3 51.9 | 49.9 51.1 54.4 | | | 51.5 (49.9-54.4) |
| S: split | 22 | 57.8 61.0 60.1 | 58.3 58.0 60.0 | 60.0 58.6 58.1 | 60.0 60.7 61.6 | 60.0 (57.8-61.6) |

Second, order A1 S5 D3 A2 S6, A = split with the placement skipped by a throwaway switch on a diagnostic build (not in this PR):

| Arm | Workers | Start 1 | Start 2 | Median (range) |
|---|---|---|---|---|
| D: default | 11, one socket | 52.7 54.3 53.1 | | 53.1 (52.7-54.3) |
| A: split, no placement | 22 | 50.5 48.7 51.3 | 50.7 56.6 56.8 | 51.0 (48.7-56.8) |
| S: split | 22 | 60.5 58.4 59.2 | 58.8 59.2 60.2 | 59.2 (58.4-60.5) |

Prompt speed 926 to 1037 tok/s across all arms of both sessions, passkey found in all 20 prompt runs of the first. VRAM hit rate 71 to 77 %.

</details>

## Misc. notes: tuning on this box

Every arm ran with the same tuning, set before this work for reasons unrelated to the split:
- `--adapt-async 1`: the adaptive tier's VRAM/RAM expert swaps happen between windows instead of inside them. On 0.1.45, 34.5 to about 54 tok/s on the LRU prompt (single runs), the swaps' cost per window from 24.5 to 2.7 ms.
- `STRATA_POOL_SPIN_US=200`, for heat: an idle worker sleeps after 200 µs instead of `kSpinBeforeSleep` 20 ms (pool.hpp:264). With ~30 ms windows and work at each of the 48 layers, the default kept every pool core spinning. No visible cost on CPU expert time (16.7 to 18.3 ms per window against 17.8).
- `STRATA_ROUTE_TAIL_SKIP=0`, to judge the quant without the lossy default skip (docs/QUANTS.md:158-162). It costs speed in every arm.
- `--resident-budget-gib 55` instead of the 40 setup.py gives on 64 GB (it leaves 24 GB beside the budget, setup.py:548): the box is headless and idles at about 4 GB, so the ~44 GiB complement fits in RAM and nothing is read from the NVMe.
- `STRATA_DECODE_TIMING=1` only adds timing log lines.
- `iommu=pt` on the kernel command line, for the main table only: Debian's kernel translated every DMA (`DMA-FQ`), which held pinned H2D at 7.7 to 9.2 GB/s on PCIe 3.0 x16; passthrough gives 12.3 GB/s and about +30 % prompt speed. It is a gen3/gen4 matter: on xrip's PCIe 2.0 x16 box (#3201) the copy speed is the same in both modes, and `iommu=pt` broke the GSP boot of a card on a patched driver there.
- A degraded inter-socket link makes the one-socket default much worse, and the split barely notices. On this box the UPI link between the two Xeons falls back to half width after some hours of uptime (machine-check log: "COR Phy lane failure, recovery in x8 width" on both ends). The main table predates it: no such error in that boot until 11 hours after the last start. After a fallback this morning, with 0.2.1 and this branch, one start per arm: the default (11 workers, one socket) gave 38.8 to 45.0 tok/s, against 56.6 and 57.4 with the split;