#2271 --shared-expert-arena: one engine loads the file, the others wait and attach

Beschreibung

**`--shared-expert-arena`: one engine loads the file, the others wait and attach**

#129 added the shared arena as a primitive and left two things to the caller: who loads the file ("no
leader/follower population protocol yet") and its lifetime. Today every engine that names the file reads the whole
pack into it again at start - 55 GiB for UD-IQ4_XS, written into pages another engine may be reading - and nothing
keeps two loads apart. This adds that part, inside the engine, so two engines of one pack (one per GPU, each with its
own expert cache and K/V; #1461's pair is one such setup) keep one RAM copy and load it once.

**What it changes** (`src/core/pinned.cu`, `pinned.hpp`, `expert_source.cpp/.hpp`, `generate.cpp`; the selftest;
`DETAILS.md`; measurements in `bench/results/2026-10-13-community-r9700-shared-expert-arena/`)

- **Who loads:** two one-byte OFD locks on the file (Linux 3.15; they belong to the open file, so the kernel drops them
  when an engine exits or dies). FILL is held by the engine that loads the file until the load passed its checks; an
  engine started meanwhile waits on it (`shared arena ...: engine N is loading it, waiting`) and then attaches - no
  read of the pack, `attached 55.43 GiB another engine loaded, nothing read`. USER is held shared while an engine maps
  the file.
- **Lifetime:** the file stays when the last engine exits, so a restart attaches too; deleting it frees the RAM. An
  engine that dies while loading leaves the file to the next one, which loads it again. A file of another pack, size or
  format is replaced when no engine maps it (USER taken exclusively, then unlinked - never truncated under a mapping)
  and refused while one does.
- **Format V2:** the header is one 2 MiB page, so the arena starts 2 MiB-aligned in the file and in memory and
  hugetlbfs can hold it (the V1 comment said it could not). V1 files are replaced like any other format when unused.
- **Pages:** the file is allocated up front (`posix_fallocate`: a tmpfs smaller than the arena is an error naming the
  size, not a SIGBUS part-way through the load), mapped before the GPU registration (`MADV_POPULATE_*`, Linux 5.14),
  and the loading engine makes its 2 MiB pages whole with `MADV_COLLAPSE` (Linux 6.1) while they are still empty. The
  start reports `N of M GiB on 2 MiB pages`; on AMD, when part of it is on 4 KB pages, it says why that matters.
- **Why 2 MiB pages matter on AMD:** `hipHostRegister` does not pin host memory on ROCm 7.14.1 (`VmPin` 0); the
  kernel's compaction moves 4 KB pages, and every move stops the process's GPU queues while the driver maps the range
  again. With 16 GiB registered, one forced compaction: 194-295 ms of stopped queues on 4 KB pages, 0-18 ms on a tmpfs
  `huge=always`; CPU reads and host->device copies were the same on both (table in `DETAILS.md`, method and the other
  backings - hugetlbfs, `hipHostMalloc`, `hipMemCreate` on the host - in the bench folder).

**What it leaves alone:** without `--shared-expert-arena` nothing changes - `PinnedArena` without a file takes the old
path, `shared_publish()` is a no-op, and `ArenaExpertSource`'s load is the same code one level deeper. The SYCL tree
keeps its own V1 copy of the option: a V1 engine refuses a V2 file (size and header do not match) rather than
misreading it, and since it takes no locks its V1 file counts as unused here - unlinked, not truncated, so it keeps
its mapping. Windows: still not implemented, as before.

**Checked**

- HIP build (gfx1201, ROCm 7.14.1): ctest 108/109, the one failure `ple_parity` as before (needs the Q2_0 model).
  `pinned_shared_test`, rewritten for the protocol, runs the other engines as child processes: the second waits while
  the first loads and attaches once it publishes; an attach in the same process; another pack and another size
  refused while mapped; a restart attaches; a file of another pack held by another process refused, replaced once
  that process is gone; a loader killed before publishing leaves the file to the next one. Run on `/dev/shm` and on a
  hugetlbfs mount (`STRATA_TEST_ARENA_DIR`): both report the file whole on 2 MiB pages. CUDA: builds and links with
  nvcc 13.4.92, sm_120, 0 errors; not run on an NVIDIA GPU.
- End to end, R9700 32 GB, Unsloth UD-IQ4_XS (55.4 GiB of experts), `--spec 4 --mtp`, 48 greedy tokens, tmpfs
  `size=58G,huge=always,noswap`:

| run | arena checksum (`STRATA_VERIFY_ARENA=1`) | tokens | start to exit |
|---|---|---|---|
| default arena (no flag) | 92a4fc0d243d7402 | reference | 105 s |
| shared file, fresh: loads it | 92a4fc0d243d7402 | same | 108 s |
| shared file again: attaches | 92a4fc0d243d7402 | same | 68 s |
| two engines 3 s apart on a fresh file (2,000-slot caches): one loads, one waits and attaches | 92a4fc0d243d7402 both | same as each other | 111 s, 109 s |

  The load itself is about 32 s here (1.7-1.8 GiB/s from the file cache); the rest of a start is the same either way.
  With the smaller caches the tokens differ from the reference from the 6th on, both engines alike - which experts a
  card holds changes the rounding, as the docs say for other cache sizes. The fresh file of the second run was on 2 MiB
  pages for 50.6 of 55.4 GiB (the machine had just freed a 55 GiB arena; `MADV_COLLAPSE` could not make the rest
  whole), the fourth for all of it after `MADV_COLLAPSE`.
- Two engines generating at the same time, one per GPU (the R9700 x16 and a second R9700 on PCIe 3.0 x4), one copy of
  the experts in RAM (69 GB of 125 GB still available with both running), 512 greedy tokens each:

| arena file | x16 alone | x4 alone | both at once, two runs: x16 / x4 |
|---|---|---|---|
| tmpfs `huge=always` | 63.3 tok/s | 42.2 tok/s | 63.9 / 41.3, 66.9 / 42.1 |
| hugetlbfs (`vm.nr_overcommit_hugepages`) | 67.2 tok/s | - | 66.8 / 37.4, 66.9 / 41.9 |

  Same checksum everywhere; each engine's tokens the same alone and in the pair. A forced compaction during the pair's
  decode stopped the x16 engine's queues 0 ms in three of four runs and the x4 engine's 0.6-0.7 s in each, on tmpfs and
  hugetlbfs alike - memory other than the arena, not traced here. The hugetlbfs pages (28,384) went back to the system
  when the file was deleted. The engine runs on hugetlbfs were made before a fix of the start line's count (hugetlb
  pages are in `Shared_Hugetlb`, not `Rss`: it said 0 of 55.44 GiB); the selftest on a hugetlbfs mount
  (`STRATA_TEST_ARENA_DIR`) now shows the file whole on 2 MiB pages.

**Not tested:** NVIDIA at run time (`cudaHostRegister` pins the file's pages there); the server (`serve/`) running two
engines (#1461's desk is one); kernels older than 6.1 (`MADV_COLLAPSE` then reports itself unavailable in the start
line).

🤖 Generated with [Claude Code](https://claude.com/claude-code)