#2177 Field report: --resident-experts` and `layer_split auto` costs nothing on 128 GB ram

本文

# Field report: the low-RAM `--resident-experts` mode costs **nothing** on a 123 GB box — even on `layer_split auto`

**Rig:** 4x NVIDIA (RTX 5060 Ti 16 GB + 2x RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM), Xeon W-1350, **123.3 GiB RAM**, NVMe.
**Model:** Qwen3.8 Flash-Next (125B MoE), GSQ-RCO **IQ3_XXS**, 262K context, `layer_split auto`.
**Engine:** 0.1.43.1. Speed only — no correctness suite (needle probes run separately, 3/3 each arm).

## TL;DR

I benchmarked  `--resident-experts` (the low-RAM mode) **on top of the winning `layer_split auto` placement**, paired back-to-back against plain mode in the same session.

**Resident mode was equal-or-faster at every context** (prefill +1.5 to +2.4%, decode inside the run-to-run band, TTFT slightly lower), while dropping the engine's RAM footprint. The resident copy was only **6.00 GiB** and every request logged **`0 blob reads from the file`** — the SSD never entered the path.


## Method

Same server, same four cards, same flags — the **only** difference between the two arms is the `--resident-experts` flag. Both arms run `layer_split auto` on the same card order. Each arm: fresh boot, 60 s settle, then 3 fresh-prompt runs (nonce per request, cache reuse 0) at 4K / 32K / 128K, plus 3 needle probes. Arms ran **back to back in one session** so box drift can't masquerade as a mode effect. Medians of 3, range shown.

Harness asserts each arm's identity before timing (model id + the `--resident-experts` flag present/absent in the engine's own `/proc/<pid>/cmdline`), and verifies the GPU is empty between arms.

## Results (median of 3, range in parentheses)

| context | metric | plain (default) | **resident (`--resident-experts`)** | resident vs plain |
| --- | --- | --- | --- | --- |
| 4K  | prefill tok/s | 1,254.1 (1,252.8–1,257.6) | **1,284.6** (1,282.8–1,284.8) | **+2.4%** |
| 32K | prefill tok/s | 3,097.4 (3,089.6–3,099.8) | **3,143.3** (3,140.6–3,154.9) | **+1.5%** |
| 128K | prefill tok/s | 3,514.9 (3,512.9–3,519.5) | **3,579.0** (3,578.2–3,584.5) | **+1.8%** |
| 4K  | decode tok/s | 113.5 (109.9–113.8) | 111.1 (107.9–112.2) | noise |
| 32K | decode tok/s | 110.3 (98.2–117.1) | 110.7 (90.7–113.4) | noise |
| 128K | decode tok/s | 103.3 (101.4–106.5) | 105.5 (105.4–108.1) | noise |
| 128K | TTFT (s) | 39.75 | **39.03** | **−1.8%** |

Prompt tokens 4,141 / 34,618 / 139,705, cache reuse 0 (fresh prompts). Needles found 3/3 in both arms. Decode run-to-run spread is ±8–15% in both arms, so decode is a tie; the prefill deltas are consistent across all three contexts and outside the per-arm prefill spread.

## What the engine actually did (resident arm)

```text
strata generate: session is up (engine 0.1.43.1)
FileExpertSource: mapped pinned cache complement ready: resident 6.00 GiB, pinned 6.00 GiB in 2 s; page-locked and mapped
strata generate: resident RAM mode: 6.00 GiB of experts in RAM (page-locked), 1536 in the GPU cache;
                 adaptive swaps exchange them with the GPU cache (no file reads)
strata serve: layer split: 98% of the experts resident, the prompt path's streamed ring 96 slots
strata serve: resident RAM: 6.00 GiB of experts in RAM, 80 exchanged with the VRAM tier, 0 blob reads from the file
```

- **Resident mode engaged, not silently downgraded to mmap** — no `does not fit` warning.
- The page-locked complement is only **6.00 GiB** (vs 7.94 GiB measured on a manual `20,24,28` split). Reason: `auto` parks 42 of 48 layers on the two 3090s where the big expert caches live (12,121 + 8,889 slots), so far fewer experts need a RAM copy. The resident mode pins only what the VRAM caches do not already hold.
- **`0 blob reads from the file` on every request** — the adaptive swaps exchange experts between the RAM copy and the VRAM cache, never re-reading the model files. This is the line that is supposed to read 0 in steady use, and it does.
- Decode expert-cache hit rate across the session: **98.6–100.0%**.
- Resident mode allowed a **larger** prompt chunk (5120 vs 4096 tokens on the plain arm) yet the prefill gain is only ~2% — consistent with the chunk not being the lever on this rig.

On previous 61 GiB build, `--resident-experts` measured **−9.1% prefill**.  With 123 GiB the flag is free (slightly positive). 

## Caveats / not measured

- Speed only; no correctness suite. Needle probes (3/3 each arm) are the only recall check here — they do **not** substitute for a long soak. A multi-hour back-to-back soak on a resident + split setup is still an open validation question.
- One parallel slot (no `--batch`), single server. Concurrency not tested.
- 123 GiB RAM box — the "free" result is specific to having headroom.
- `--resident-experts` on a split exercises the adaptive-swap path; if output ever looks wrong, `--adapt-every 0` disables the tier and the answer must match plain mode token-for-token.

## Repro

`~/bin/flash-next`-launched serve on one port, `layer_split auto`, card order giving CUDA0=3060 / CUDA1=5060 Ti / CUDA2+CUDA3=3090s. Two arms differing only by `--resident-experts`, run back to back, 3 reps at 4K/32K/128K + 3 needles each.