#1958 [Field Observation] Huge Pages does not improve performance

Description

# Field observation — huge pages: the hugetlb arena works, and it changes nothing measurable (0.1.42, 4-GPU rig)

Posting as an observation. No bug to fix, no regression — the huge-page path is doing exactly what the header in `include/strata/core/pinned.hpp` says it should, and on this machine that turns out not to matter.

**Rig:** RTX 5060 Ti 16 GB (CUDA0) + RTX 3090 24 GB (CUDA3) + 2× RTX 3060 12 GB (CUDA1, CUDA2), no NVLink. Three of the four cards run PCIe 3.0 **x4** (engine probes 3.1 GB/s, `pcie_frac 0.09`); the 5060 Ti is x8 (14.0 GB/s, `pcie_frac 0.38`). 61 GiB host RAM (`MemTotal 64249528 kB`), single NUMA node.

**Model:** Qwen3.8-Flash-Next, **IQ3_XXS**, 48 layers, `--max-context 262144 --kv q4_0`, `--expert-cache auto`, `--prefill auto`, `--spec 4 --spec-min-p 0.5`, `--mtp`, `--vision`, `--vram-reserve-mib 700`, `STRATA_PF_FUSED=1`.

**Engine:** `strata 0.1.42` (binary built 2026-10-10 16:50).

**Layout, both arms:** `"gpu": [0,1,3,2]`, `"layer_split": "20,24,28"` → layers 0-19 (5060 Ti) / 20-23 / 24-27 (the 3060s) / 28-47 + head (3090). Caches 6352 / 2048 / 2048 / 8827 slots, **19275 of 24576 profiled pairs resident (78%)**, prompt chunk **6144** with the usual #448 clamp note.

## What I ran

One variable: `vm.nr_hugepages`, set to **20480** (40.0 GiB of 2 MiB pages; the arena asks for 20464) while the engine was down, and released again afterwards. The pool has to be reserved with the engine stopped — 40 GiB does not fit next to a running engine — and `MAP_HUGETLB` is all-or-nothing, so a pool that comes up short fails the mapping rather than half-working.

Same config, same split, one benchmark session, two measured boots:

| arm | `vm.nr_hugepages` | prefill tok/s | run spread | gen tok/s | engine RSS | MemAvailable |
| --- | --- | --- | --- | --- | --- | --- |
| no pool | 0   | **950.5** | 920.0–951.4 | 99.2 | 45.4 GiB | 8.8 GiB |
| hugetlb | 20480 | **950.0** | 920.3–950.6 | 98.6 | **5.4 GiB** | 8.9 GiB |

Harness: cold salted ~8.4k-token prefill (`max_tokens 1`, temp 0) + 512-token gen, 3 runs × 3 reps, median of 9. Run-to-run spread inside one arm on this rig is ~1.3% prefill / ~7.9% gen, so **−0.1% prefill and −0.6% gen is noise**. Both arms PASS their own startup checks.

## The mapping really happened

Boot 1, 09:24:41 (no pool) — log line 26155:

```
strata generate: expert arena read through the file cache (buffered (not Windows))
strata generate: expert arena: cudaHostRegister PORTABLE ok; MAP_HUGETLB unavailable (needed 20464 2 MiB pages, vm.nr_hugepages=0); transparent huge pages requested (MADV_HUGEPAGE)
```

Boot 2, 09:29:23 (pool reserved) — log line 26334:

```
strata generate: expert arena read through the file cache (buffered (not Windows))
strata generate: expert arena: cudaHostRegister PORTABLE ok; hugetlb 2 MB pages
```

Boot 3, 09:32:47, same config, pool released again — log line 26514, back to `MAP_HUGETLB unavailable ... vm.nr_hugepages=0`. So the flip tracks the pool and nothing else.

Corroborating host state during the hugetlb arm:

- engine RSS **45.4 GiB → 5.4 GiB** — the 39.97 GiB arena left normal RAM and lives in the pool.
- `cudaHostRegister PORTABLE ok` on a hugetlb mapping, i.e. pinning the 2 MiB pages worked fine.
- nothing swapped: 328 KB of the 32 GiB swap file in use after the run, and the engine held 5.4 GiB RSS while serving all 9 harness runs.
- `AnonHugePages: 0` while the default-mode engine is running: the fallback's `MADV_HUGEPAGE` request does not leave 2 MiB pages behind, because the arena is `cudaHostRegister`-pinned. So the no-pool arm genuinely ran on 4 KB pages — this was a real 4 KB vs 2 MB comparison, not THP vs hugetlb.

## Why it did nothing here

1. **78% of the routed expert mass is already in VRAM** (19275 of 24576 pairs). The CPU pool only touches the streamed remainder, so arena TLB reach is not on the critical path in either prefill or decode.
2. Prefill on this rig is **placement-bound**: it moved +15.5% when I moved layers 16→20 onto the x8 card, and it sat at ~950 in both of these arms regardless of page backing. Chunk stayed 6144 in both.
3. What *was* paid is real: 40 GiB locked out of the page cache for the whole session (the engine itself measured 14.00 GiB of normal RAM available on that boot). A TLB gain and a colder file tier would have cancelled inside the same 1.3% band, and this arm alone cannot separate "there was no TLB cost to begin with" from "gain offset by page-cache loss". Given 1 and 2 I read it as the former, but I did not measure it directly.

**Practical conclusion for this class of machine: don't turn huge pages on.** Zero measured gain, 40 GiB of RAM removed from the file cache, and a startup line that reads like an imminent OOM (below).

## Scope note: the arena only exists in the default mode

Worth a line in the docs, because it decides where huge pages are a lever at all. With `--resident-experts` there is **no pinned arena**:

```
grep -ac "arena" strata-iq3_xxs.log          -> 241   (default mode, many engine boots)
grep -ac "arena" strata-iq3_xxs-lowram.log   ->   0   (--resident-experts)
grep -ac "arena" strata-iq3_s.log            ->   0   (--resident-experts)
```

In the resident mode the engine builds a page-locked cache complement instead (`FileExpertSource: mapped pinned cache complement ready: resident 13.21 GiB, pinned 13.21 GiB in 10 s; page-locked and mapped`) and reports `0 blob reads from the file`. `pinned.cu`'s `MAP_HUGETLB` path never runs, so huge pages are a **default-mode-only** lever.

Related: `STRATA_NO_LARGEPAGES=1` is documented as the same-run A/B switch, and it is a good idea, but it cannot be used as the control on a box this size — a 40 GiB reserved pool *plus* a 40 GiB 4 KB-page arena needs ~86 GiB of a 61 GiB machine. The only safe comparison here was pool-off vs pool-on across two boots.