#1866 [Feature]: --resident-budget-gib on a layer split — fit UD-IQ4_XS's 55 GiB of experts on 46 GB RAM + 2 GPUs

Descrição

### Goal

Let the Unsloth RAM-budget mode (`--resident-budget-gib`, the FileExpertSource path) run on a
layer split (`--gpus`), so several GPUs' VRAM can hold the expert cache when the experts don't
fit RAM. Today the engine refuses the pair (exit code 2, #498), so a budget config is
single-GPU and the other cards' VRAM sits idle.

### Measured on this box (0.1.41): neither mode alone fits the model

Machine: 2× RTX 2080 Ti 22 GB (44.5 GB VRAM total), 46 GiB usable DDR4, NVMe.
Model: Qwen3.8-Flash-Next **UD-IQ4_XS** (87.3 GB download, **55.43 GiB of experts**).

- **Budget mode (single GPU)**: one card fits only **10.82 GiB** of expert cache (4802 slots)
  after dense weights + 262K q4_0 KV + vision + draft head; GPU 1's 21.5 GB idles. With
  `--resident-budget-gib 37` (the safe ceiling here; 40 leaves ~2 GB of MemAvailable), the tiers
  are 10.82 + 37 = **47.8 < 55.43** → ~7.6 GiB of experts live only on the SSD. The VRAM-tier
  hit rate is fine (79–85%), but a cold-topic request reads 11–15 GB from storage (decode waits
  8–10 s per request), so decode is 16–21 tok/s and prefill 525 tok ≈ 6.1 s.
- **Same box, IQ3_XXS** (40 GiB experts, dual-GPU mmap mode): combined expert cache **27.7 GiB**
  (13.8 main + 13.9 helper), experts.bin fully page-cached → decode 62–92 tok/s, prefill 525 tok
  1.5 s.
- **Counterfactual**: budget + layer split would give 27.7 (dual-GPU cache, measured here with
  IQ3_XXS) + 37 (RAM) = **64.7 ≥ 55.43 → zero file-tier reads** on a 46 GB machine. Today
  neither current mode gets there alone.

0.1.40 already allowed `--resident-experts` on a split (#642) — the Unsloth budget path looks
like the last mode that is single-GPU only. For RAM ≥ 64 GB the single-GPU budget fits
(48 + 10.8 > 55.43); this ask is about the 46–48 GB class with several GPUs.

### Workaround today

Budget mode on one GPU and accept the SSD tail, with a MemAvailable watchdog beside it; or mmap
mode and hope the page cache covers the working set (it can't here: 55.4 GiB vs ~41 GiB).

### Related safety note

On this machine, the no-budget path auto-chosen for a UD model on two GPUs (all experts resident
+ partial mlock) hard-froze a 47 GB PC during load — mlock pages are unreclaimable, so the OOM
killer couldn't even fire. Setup's warning (#737) helps, but an engine-side floor check (expert
bytes vs RAM) with a clean error would save someone's session.

Happy to test any build on 2×2080 Ti / 46 GB and report numbers.

---

### 中文版(同一内容)

**目标**:让 Unsloth 档的 RAM 预算模式(`--resident-budget-gib`)可以与 layer-split(`--gpus`)同用——专家总量超过内存时,用多卡显存一起装专家缓存。目前引擎拒绝该组合(exit code 2,#498),budget 配置只能单卡,其余显卡显存闲置。

**实测**(0.1.41;2× RTX 2080 Ti 22GB,46GB 可用内存,UD-IQ4_XS,专家 55.43 GiB):

- budget 单卡:扣除稠密权重 + 262K q4_0 KV + 视觉 + draft head 后,单卡专家缓存仅 **10.82 GiB**(4802 槽),另一张卡 21.5 GB 闲置;`--resident-budget-gib 37`(本机安全上限,40 时 MemAvailable 只剩 ~2G)下 10.82 + 37 = **47.8 < 55.43**,~7.6 GiB 专家只能落 SSD。显存层命中率 79–85% 正常,但冷话题每条请求盘读 11–15 GB(decode 每条等盘 8–10 秒)→ decode 16–21 tok/s,prefill 525tok 6.1s。
- 同机 IQ3_XXS(专家 40 GiB,双卡 mmap 模式):合计专家缓存 **27.7 GiB**(主 13.8 + helper 13.9),experts.bin 整个进页缓存 → decode 62–92 tok/s,prefill 1.5s。
- 反事实:若 budget 支持 layer-split,27.7 + 37 = **64.7 ≥ 55.43,零盘读**——46GB 内存就能跑起 UD-IQ4_XS。目前两种模式单独都装不下。

0.1.40 已允许 `--resident-experts` 上 layer-split(#642);Unsloth budget 路径应该是唯一还锁单卡的模式。内存 ≥64GB 时单卡 budget 就够(48+10.8 > 55.43),这个请求针对的是"46–48GB 内存 + 多张卡"这一档。

**现用绕法**:budget 单卡 + 接受 SSD 尾巴,旁边挂 MemAvailable 看门狗;或 mmap 模式赌页缓存盖住工作集(本机盖不住:55.4 vs ~41G)。

**附带安全建议**:本机曾因 UD 模型在双卡上无 budget 被自动选成"全专家驻留 + 部分 mlock",加载阶段整机锁死(mlock 页不可回收,OOM killer 都无法生效)。setup 的 #737 提示有帮助,但引擎侧加一道"专家量 vs RAM"底线检查、干净报错,会更安全。

愿意在 2×2080 Ti / 46GB 上帮忙测任何构建并回报数据。