#3185 [Engine] arena: hand back the block for layers a GPU holds whole (opt-in, saves 21.7 GiB of RAM)

说明

## Title

Issue: Resolves # (no issue filed — this grew out of a private investigation on a 3090 + V100 rig; happy to open one if you prefer the change tracked there)

## Summary

On a **layer split**, the layers a GPU stage holds *whole* — every one of its experts resident in that card's cache — are never read from the CPU expert arena. Today they are still copied into the arena and paid for in RAM/commit for the whole run. This adds an opt-in path that loads those layers into a **separate `PinnedArena`** and hands the whole block back to the OS once the residency table proves the claim.

Enabled with `STRATA_ARENA_RELEASE_BLOCK=1`. **Default off: one block, byte-identical to today.**

```
strata generate: arena-skip: layers 27-47 are fully resident: 21.71 GiB of the expert arena handed back to the OS
```

Why a second allocation and not `madvise` / `MEM_DECOMMIT`: `release()` already exists and is already called for every VRAM-resident pair, but it only acts on the **file-backed** arena (`STRATA_ARENA_MMAP`), where dropping a page just means faulting it back from the file. The default arena is anonymous **2 MiB large pages**, so (a) a range cannot be decommitted — `MEM_DECOMMIT` on a `MEM_LARGE_PAGES` allocation fails, the region must go back whole — and (b) a released anonymous page reads back as **zeros**, i.e. silent corruption. Making the releasable layers their own allocation is what keeps the large pages.

## What changed

| File | Change |
|---|---|
| `include/strata/core/pinned.hpp` | `load_experts_ranges`: `dst_bias` (the file offset that maps to `dst[0]`) |
| `src/core/pinned.cu` | the write path honours `dst_bias` (`dst + off - dst_bias`) |
| `include/strata/core/expert_source.hpp` | `load_experts_gguf`: `dst_bias` **and** a layer range `[l0, l1)`. `ArenaExpertSource`: `set_split_layer()`, `split_layer()`, `release_tail()`, `tail_released()`, and the tail block's members |
| `src/core/expert_source.cpp` | the second `PinnedArena`, its load (`dst_bias = boundary_off_`), the `blob_at()`/`blob()` branch, `release_tail()` (one `MEM_RELEASE`), `close()` |
| `src/program/generate.cpp` | `set_split_layer(split_at.front())` behind the switch; after `host_res` is built, the per-layer "all 512 experts resident" check and the release, with the log line |

`load_experts_gguf` needed **two** things, not one: it wrote at `dst + lay.blob_offset(...)` (a second allocation cannot be served by that) **and** it walked all `lay.n_layers`, so a block would otherwise fill all 48 layers.

## Extra Notes

**Why it is safe.** The block is released only when *every* pair of those layers has `host_res != kNotResident`, i.e. all 512 experts of all 21 layers sit in the V100's cache. A layer with all 512 experts resident has **no spare slot** to `admit()` another one, so the adaptive tier can never evict one of them; hence no CPU-pool read can ever land in the released block. And if that reasoning were ever wrong, `blob_at()` answers **nullptr** for those layers, which the pool turns into `d.failed = "the expert source could not produce a blob"` — a loud error, not zeros.

**Why the release waits for `host_res`.** Sizing it up front from free VRAM was rejected: on the test machine the V100 has only **0.26 GiB** of headroom over its layers' experts (21.70 needed vs ~21.96 available after the prompt buffers, KV residency and the reserve), so an estimate that is 1 % off would release a layer that can still miss. `host_res` is the first point where the residency is known exactly.

**Concretely, on the machine it was written for** — RTX 3090 (24 GB, x8) + Tesla V100 (32 GB, x8), Qwen3.8-Flash-Next GSQ-RCO **IQ3_S**, 48 layers × 512 experts = 24,576 pairs, `--max-context 262144`, `--kv int8 --kv-resident 32768`, `--prefill 4096`, `--pipeline-windows 2`, `--spec-min-p 0.7`, `--conversation-cache-mib 4096`, Windows 11. `--layer-split auto` prices decode and picks **K=27** (best of 46 placements; K=31 is a 0.2 % near tie):

| | RTX 3090 (CUDA0) | Tesla V100 (CUDA1) |
|---|---|---|
| layers | 0–26 (27) | 27–47 (21) + head + draft |
| decode cost per layer | 0.48 ms | 0.66 ms |
| experts those layers hold | 27 × 512 = 13,824 | 21 × 512 = 10,752 |
| expert cache slots | 8,516 (15.46 GiB) → **62 %** | **10,752 → 100 %** |
| free VRAM once loaded | 1,413 MiB | 1,676 MiB |

So 21 × 512 × 2.07 MiB = **21.70 GiB** of the 46.84 GiB arena that the CPU can never read.

| Before / after, same config, same model, same boot | before | after |
|---|---:|---:|
| `experts loaded` | 46.84 GiB | 46.84 GiB (unchanged) |
| **Windows commit free once loaded** | **0.5 GiB** | **21.4 GiB** |
| process commit | ~64.5 GiB | **~43.4 GiB** |
| VRAM hit / PCIe / CPU pool | 96.15 / 0.25 / 3.55 % | 96.70 / 0.20 / 3.10 % |
| decode tok/s (median, 10 replayed real agent requests) | 110.2 | 106.5 |
| first token (median) | 2.36 s | 3.03 s |
| `file_blobs` / `file_mb` | 0 / 0 | 0 / 0 |

The 0.5 GiB figure is why this exists: the engine warns `the Windows commit limit is 116.7 GiB and 2.9 GiB of it is left ... it can end at the limit with no message (exit code 0xC0000409)` on that machine, and a released block is the difference between "at the limit" and 21 GiB of headroom.

**On the decode / first-token deltas above** — they are not the mechanism. The released block is never touched on the decode path (proved by `file_blobs = 0` and zero errors), the two runs differ in file-cache state (the run after a fresh start pays a cold 204K-token prompt), and per-block deltas even change sign (+1.8 % / −6.3 %). A paired comparison of the same 10 requests against the 0-release arm gives **−1.2 % decode, 5 up / 5 down** = noise. 20 replayed requests: `could not produce a blob` **0**, `file_blobs`/`file_mb` **0** on every request.

**Where it works.** Both expert sources: the pack above has **no `experts.bin`** (`native_experts.txt` → the GGUF shards), so `open()` takes the `from_gguf` path. Not touched: `--mmap-experts` / `--resident-*` (they already have their own partial-residency machinery), the padded `STRATA_PIN_BY_NEED` layout, `--shared-expert-arena`, `--expert-cache-device1..3` — `open()` falls back to one block for all of them.

**Build / verify.** `build-cuda12`, `strata` target, `-DCMAKE_CUDA_ARCHITECTURES="70;86"`, exit 0. Smoke-tested on this rig: `session is up (engine 0.1.45)`, the release line printed, K stayed 27, and the same 10-request replay ran clean.

**Note on the base.** This branch was cut from `477ac13c` and so predates the `pin_reserve_*` sequence that now wraps `arena_src.open(...)`; that is why GitHub reports it as not mergeable against today's `main`. Rebasing is a one-place edit (keep the new `pin_reserve_take()/pin_note_arena()/pin_reserve_release_soft()` trio and insert the gated `set_split_layer()` before it) — say the word and I will rebase rather than have you resolve it.

**Design note** is in the same PR: `docs/ARCH-EXPERT-MEMORY-2026-10-19.md` — the 94 % / 96 % hit-rate breakdown, the per-miss cost of the three paths (PCIe 0.15 ms → 3090, 0.31 ms → V100, CPU pool ~1 ms), and why only the 512/512 layers qualify.