#384 Layer split with the low-RAM resident mode: 2-GPU numbers on 16+16 GB / 47 GB RAM (IQ3_XXS and Q2_0)

Description

Follow-up to #250 (closed). The answer there was that the low-RAM resident mode runs on one GPU by design. #364 describes a related workaround. These are measurements from a 2-GPU / 47 GB RAM box to help decide whether the setup should offer the layer split here.

### Setup
- 2× RTX 5060 Ti 16 GB, PCIe 5.0 x8 each (~26–28 GB/s H2D when the link is healthy)
- i9-12900KF (AVX2, no AVX-512), 47 GB RAM, Ubuntu, Strata 0.1.30 (30ec18e)
- `--max-context 131072 --kv int8 --spec 4 --mtp`, `--expert-cache auto`
- CPU package power capped at 45 W (temporary, stock box cooler). "Uncapped" runs hit 100 °C and throttle, so treat them as approximate.

Bench: two short prompts (27 tokens → 150–180 generated), then cold 27k and 46k prompts (128 generated). Numbers come from the `strata serve: prompt …` log lines.

### Results

| Config | GPUs | Short decode | 27k prefill / decode | 46k prefill / decode |
|---|---|---|---|---|
| IQ3_XXS, setup default (`--resident-experts`, low-RAM), CPU uncapped | 1 | 39–50 t/s | 1,573 t/s | 1,060 t/s |
| IQ3_XXS, setup default, CPU 45 W | 1 | 30–37 t/s | 966 t/s | — |
| **IQ3_XXS, hand-edited json** (`"gpu": [0,1]`, `"layer_split": "auto"`, `--resident-experts` removed), CPU 45 W | 2 | **54–61 t/s** | **1,090 / 58 t/s** | **1,340 / 65 t/s** |
| Q2_0, setup `--gpus 0,1 --low-ram off` (layer split 0–24 / 25–47), CPU 45 W | 2 | 66 t/s | 1,338 / 62 t/s | 1,760 / 76 t/s |
| Q2_0, same, CPU uncapped | 2 | 68 t/s | 1,500 / 80 t/s | 1,746 / 70 t/s |

Caveat: this host has an unrelated PCIe problem where H2D drops to ~7 GB/s after a load transition. It happened during the 2-GPU runs: the engine probe logged `CUDA1 PCIe probe 7.4 GB/s` on the IQ3_XXS 2-GPU start, and a measurement after the Q2_0 runs showed 6.9 GB/s. So the 2-GPU numbers are probably a lower bound.

### Observations
1. **On a 16 + 16 GB pair, the layer split roughly doubles decode.** IQ3_XXS goes from 30–37 to 54–61 t/s at the same 45 W cap. With one GPU, the experts that miss the cache run on the CPU (AVX2), so decode and prefill follow the CPU power limit. With two GPUs, the second card's expert cache covers ~98% of the routed mass, which removes most of that dependence.
2. **The engine still refuses resident + split.** With `--resident-experts` and a layer split it exits with `--resident-cpu-experts does not support layer splits or remote expert caches`. Without the flag it runs, but RAM sits at 46/46 GB used and 0 available during long prompts. That works for a bench, but it is fragile next to anything else on the box.
3. Q2_0 with a layer split is the fastest config here and stays within RAM (37 GB used), but it is the lower-quality quant.

### Request
- **Engine:** allow `--resident-experts` (or a variant of it) together with the layer split, so the split IQ3_XXS config can run with predictable RAM instead of filling page cache to 0 available.
- **Setup:** for 2-GPU machines where the low-RAM rule (RAM < arena + 10 GB) applies, offer the 2-GPU layer-split variant instead of silently dropping to one GPU, as #364 also suggests. At minimum, print that the json edit is an option and what it costs in RAM.

Happy to rerun anything on this box (including 0.1.31) if useful.