#1837 [Feature]: resident RAM mode — let the RAM copy follow the conversation when it cannot hold every expert (opt-in, PR #1838)

本文

### What do you want to do?

Make decode faster on a PC whose RAM cannot hold every expert the GPU does not.

In the resident RAM mode the RAM copy is chosen once, at start, from the expert profile. Experts outside it are read
through the file tier when needed, but are not retained in the resident copy. The adaptive tier exchanges the resident
copy and VRAM. So frequently needed experts can remain on the file tier while RAM places hold experts the conversation
no longer uses.

**Proposal:** let the RAM copy follow the conversation as the VRAM tier does (opt-in, `STRATA_RAM_ADAPT=<n>`). The
implementation, a unit test and all the measurements are in **#1838**.

RTX 3060 12 GB, 32 GB of RAM, Linux, IQ2_XS (about 6,000 of the 24,576 experts on the file tier), measured again
against v0.1.42, `--adapt-async 1`, `--spec 3`; decode tok/s, median of three fresh-process runs per arm:

| route tail skip | v0.1.42 | `STRATA_RAM_ADAPT=8` | |
|---|---:|---:|---:|
| 7 (the 0.1.42 default) | 45.19 | 49.51 | **+9.6%** |
| off (`STRATA_ROUTE_TAIL_SKIP=0`) | 35.36 | 40.40 | **+14.3%** |

Prompt speed stays within run variation (466-471 tok/s without, 470-472 with). The v0.1.41 numbers this issue was opened with
(+11.9% to +17.3% over four settings) are kept in #1838.

**Compared with `STRATA_ROUTE_TAIL_SKIP`.** `STRATA_ROUTE_TAIL_SKIP=7` omits reads and computation for eligible
experts outside VRAM; RAM residency does not exempt them from skip. `STRATA_RAM_ADAPT` instead retains frequently
needed file-tier experts in the resident RAM copy. With default tail skip, decode file reads per generated token
still fall from 11.4 to 4.7, so the gain remains.

The four prompts of a run are in four languages, one after another. Decode tok/s per prompt (`--adapt-async 1`,
`--spec 3`, the default tail skip, mean of the three runs):

| | English | Japanese | Chinese | German |
|---|---:|---:|---:|---:|
| v0.1.42 | 48.1 | 46.4 | 44.1 | 42.7 |
| `STRATA_RAM_ADAPT=8` | 49.8 | 47.4 | 50.7 | 50.2 |

**How:** before a decode window, while no adaptive round is in flight, a file-tier expert the windows keep routing to
(the adaptive tier's own decayed count, 2 or more) takes the RAM place of an expert of its own layer that is not routed
to (count below 0.5); at most `n` per window. The RAM stays the same size; the routing, the weights and the VRAM tier's
rule are untouched, and nothing runs when the variable is unset (checked: same answer hash as v0.1.42).

**Related:** #1324 (`--resident-lru-gib`) answers the same observation with a separate LRU pool inside the RAM budget.
Built and measured on this machine with the same workload and total RAM, its branch gives +7% / +10% / +12% decode with
a 1 / 2 / 4 GiB LRU against its own baseline (measured before v0.1.42, not repeated); the numbers are in #1838.

**Not tested:** AMD model inference and HIP `--rotation-gpu`, SYCL and Windows builds, other quantizations, and RAM
sizes other than 32 GB. This feature does not run with a layer split, peer/helper GPU or `--batch`.
`docs/TEST_REQUESTS.md` asks for results from a Linux host with 32 GB of RAM or less; this is one.