#2684 [Performance]: NUMA-local binding on a dual-socket PC gives +37-40% prompt throughput (GTX 1080 Ti, 2x Xeon E5-2696 v4)

Beschreibung

## What happened

On a dual-socket PC, the engine's pinned expert arena and its CPU pool spread across both NUMA nodes by first-touch. The GPU sits on one socket, so roughly half of its DMA and half of the CPU expert reads cross the inter-socket link. Binding everything to the GPU's node made prompt throughput **37-40% faster** and first-token times **27-32% shorter**, with no engine change.

## Hardware

- GPU: **NVIDIA GeForce GTX 1080 Ti (GP102, sm_61), 11 GiB GDDR5X, PCIe Gen3 x16** — on **NUMA node 1**
- CPU: **2x Intel Xeon E5-2696 v4** (22 cores / 44 threads each, 44c/88t total, Broadwell-EP, **no AVX-512**)
- RAM: 125.8 GiB (64 GiB per node); NVMe SSD
- Expert cache on this card: **1,513-1,514 slots** (hit rate 16-42%), so **58-84% of routed experts are computed on the CPU pool (AVX-2)** — the CPU side dominates decode
- A tiny GT 710 handles the display on NUMA node 0; the 1080 Ti at `84:00.0` does the compute

## The change

```bash
numactl --preferred=1 --cpunodebind=1 python serve/server.py ...
```

- `--preferred=1`: the 46.84 GiB pinned expert arena + 3.09 GiB pinned K/V land on node 1 (the GPU's node). Node 1's 64 GiB holds it beside ~6 GiB of system use.
- `--cpunodebind=1`: the expert pool's workers run on node 1's 22 cores (44 threads). This halves the available cores (22 vs 44 spread) — and is still **faster**, because the 44-core spread paid the 2.1x remote-memory penalty (NUMA distance 10 vs 21) on half its reads.
- Additionally we moved 184 non-Strata processes to node 0 and reserved node 1 for the engine (`AllowedCPUs=22-43,66-87` in a systemd drop-in), so the desktop does not steal cycles from the pool.

Also on this box: `vm.nr_hugepages` was 24000 (46.9 GiB reserved for MAP_HUGETLB). The engine needs 23,983 pages but found 8,221 already reserved, so MAP_HUGETLB failed every start and fell back to THP while the idle pool was invisible to normal allocations — the fallback load then tripped systemd-oomd (the unit was killed and restart-looped). Setting `vm.nr_hugepages = 0` removed the trap; the engine loads with THP + regular pages.

## Measured (the same suite, same card, same day)

| Metric | before (no binding) | **after (preferred=1 + cpunodebind=1)** |
|---|---:|---:|
| 4K prompt, fresh | 202.4 tok/s (201.1-202.8) | **276.3 tok/s (273.1-280.0)** (+36.6%) |
| 16K prompt, fresh | 194.6 tok/s (193.5-196.7) | **272.6 tok/s (272.3-277.8)** (+40.1%) |
| 4K first token | 19.92 s | **14.62 s** (-26.6%) |
| agent turn first token | 25.60 s | **17.47 s** (-31.6%) |
| chat decode | 27.4 tok/s | 24.9 tok/s (within noise) |
| agent decode | 23.6 tok/s | 21.5 tok/s (within noise) |
| agent cache hit | 66.0% | 68.8% |

Speeds are the engine's own timings; first-token times are client-side over HTTP; `tools/strata_bench.py` suite, 3 runs per test. Decode is unchanged because it is bound by the AVX-2 expert compute itself (local now, but the same 22 cores), not by memory locality.

The PCIe link (Gen3 x16, probe 11.6-11.9 GB/s) carries only ~200-257 MB/s during prompts — the interconnect is not the limiter either way; the win is the CPU-side reads hitting local DDR4 (4 channels ≈ 68 GB/s) instead of UPI (~19 GB/s shared).

## What this suggests for setup

A dual-socket PC whose GPU is on one socket gains from a start-time binding to that socket. Setup could read the GPU's `numa_node` sysfs attribute and, when it is >= 0 and the node has RAM for the arena, mention or apply the equivalent of `numactl --preferred=<gpu_node> --cpunodebind=<gpu_node>`. The engine already knows the arena size and the CPU topology. A `--numa auto|off|<node>` knob would make this a first-class option instead of a wrapper script.

## What was not tested

- The CPU-only side in isolation (binding memory but not CPUs, or vice versa) — we changed both at once and measured the combination.
- Whether `--cpunodebind=1` alone hurts on a PC whose GPU is on node 0 (the mirror case).
- AMD/EPYC (different interconnect and distances).
- Long-run stability (thermal, cache drift) under the binding — one suite, ~11 minutes.
- The hugepage trap's fix (`nr_hugepages=0`) on a PC that actually needs MAP_HUGETLB.

🤖 Generated with [Claude Code](https://claude.com/claude-code)