#2824 [Linux, 2x RTX 3060, 32 GB RAM, SATA SSD, 0.1.44] `--prefill auto` picks 8192 by VRAM alone; lent slots + file tier exceed the page cache, so a 28.7K prompt re-reads 28-45 GB (TTFT 164-287 s); `--prefill 4096` fits: 0.9-1.4 GB read, TTFT 36-38 s

Description

**Strata version, GPU, OS:** `0.1.44` (ed4773eb, built from source with `STRATA_PORTABLE=ON`, CUDA 13.0, sm_86) · 2x NVIDIA GeForce RTX 3060 12 GB, layer split auto (K=39), CUDA0 at 01:00.0 Gen3 x16, CUDA1 at 02:00.0 Gen2 x4 (chipset slot) · Linux 6.8.0-147-generic, Intel i7-4790K (4C/8T, no AVX-512), 32 GB RAM (MemTotal 32,043 MB), 8 GiB swap, one PNY 250 GB SATA SSD (ext4, model and swap on it) · NVIDIA open kernel module 580.178.

**Model and flags.** Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS, native pack, GGUF shards in place (36.53 GiB + 26.82 GiB, no experts.bin):

```
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt> --max-context 32768
--kv int8 --kv-resident 32768 --resident-budget-gib 12 --layer-split auto
```

Every run below is the same binary and the same three prompts: a 7,718-token warm-up as request 1 on a fresh engine, nine ~105-token requests, then a 28,683-token prompt (repeats use 28,679 and 28,709 tokens), 256 completion tokens each. Only `--prefill`, `--resident-budget-gib` and `--ple-io` change between rows.

### Summary

With `--prefill auto` the engine picks an 8192-token chunk and lends the prompt path 5,532 GPU cache slots (its own line: `the split stages lend 5532 slots to the prompt path: 0 keep their experts in RAM too`). Those lent experts are read from the GGUF on every chunk. On this box the bytes a long prompt reads from the files (the experts outside the GPU cache and the RAM copy, plus the lent slots) are larger than the RAM left for the page cache, so the set cycles through the cache and the drive serves 28-45 GB per 28.7K-token prompt. `--prefill 4096` lends 2,966 slots, the set fits, and the same prompt reads 0.9-1.4 GB once the cache is warm. The auto scan only tests VRAM (slot count and the lend percentage); it does not look at RAM. The one RAM-aware clamp in the tree is the gfx12-only #2090 step from 16384 to 8192.

### Measurements

Long prompt, time to first token and bytes read from the drive during the request (`/proc/diskstats` sectors read on `sda`, sampled at the stamps before and after each request; swap is on the same drive, so swap-in is included, and the sector deltas run 1-7% above the engine's own storage counter. `pswpin` sampled per request: 4-84 MB per rep in the five fixed-chunk cells, except 0.58 GB on rep 1 of the 4096/12 and 6144/12 cells; 0.73-0.93 GB per long prompt on the `auto` rows (2.3 GB across each `STRATA_LOOKAHEAD=0` cell's three), about 2-3% of their 28-45 GB):

| `--prefill` | budget GiB | PLE | rep 1 (cold) TTFT | reps 2-3 TTFT | prefill tok/s, reps 2-3 | drive GB per prompt, reps 1 / 2 / 3 | cell |
|---|---|---|---|---|---|---|---|
| auto (8192) | 12 | direct | 277.0 s | | 104 | 40.2 | `n4d-d-auto-s10-1016T144018` |
| auto (8192) | 12 | direct | 189.6 s | | 151 | 31.1 | `...-drop-1016T145417` |
| auto (8192) | 18 | direct | 171.3 / 182.9 / 236.7 s (3 starts) | | 121-167 | 29.3-31.6 | `...-b18-*` |
| auto (8192) | 12 | direct, `STRATA_LOOKAHEAD=0` | 276.0 s | 204.6 / 196.8 s | 140-146 | 37.7 / 34.4 / 39.5 | `...-nola-1016T151848` |
| auto (8192) | 12 | direct, `STRATA_LOOKAHEAD=0` | 200.8 s | 163.9 / 286.8 s | 100-175 | 27.9 / 28.5 / 45.2 | `...-nola-r2-1016T160734` |
| 2048 | 12 | direct | 69.3 s | 53.0 / 50.9 s | 542 / 565 | 5.7 / 0.8 / 0.6 | `...-pf2048-reps-1016T195949` |
| 2048 | 12 | mmap | 69.9 s | 54.5 / 51.3 s | 526 / 560 | 6.3 / 0.9 / 0.6 | `...-pf2048-mmap-reps-1016T203603` |
| 4096 | 12 | mmap | 91.0 s | 38.2 / 37.4 s | 751 / 768 | 8.9 / 1.4 / 0.9 (rep 1 includes 0.58 GB swap-in) | `...-pf4096-mmap-reps-1016T205114` |
| 6144 | 12 | mmap | 107.3 s | 40.0 / 35.2 s | 718 / 816 | 12.5 / 2.8 / 1.2 (rep 1 includes 0.58 GB swap-in) | `...-pf6144-mmap-reps-1016T210420` |
| 4096 | 16 | mmap | 59.1 s | 37.4 / 36.1 s | 767 / 796 | 7.1 / 1.2 / 1.0 | `...-b16-pf4096-mmap-reps-1016T211559` |

The 7,718-token warm-up as request 1: 126-198 s on the five `auto` starts (198.5, 197.1, 170.9, 146.4, 126.4), 103.5 s at 4096/12, 54.7 s at 4096/16.

Across 11 long prompts on seven `auto` starts at two budgets, TTFT is 164-287 s and the drive serves 28-45 GB per prompt. At 2048, 4096 and 6144 the steady state is 35-55 s and 0.6-2.8 GB. The budget does not change which chunk fits: at `auto` the drive serves 28-45 GB per prompt at budget 12 and 29-32 GB at 18, and at 4096 it serves 0.9-1.4 GB at 12 and 1.0-1.2 GB at 16.

### What the engine lends per chunk

Read off the `lend sizing` line of one start per chunk (`n4d-lend-pf2048-1016T195329`, `n4d-lend-pf4096-1016T195628`, the 6144 and auto cells above):

| chunk | slots lent | GiB (at the engine's 7.54 GiB for 5,532) |
|---|---|---|
| 2048 | 1,688 | 2.3 |
| 4096 | 2,966 | 4.0 |
| 6144 | 4,246 | 5.8 |
| 8192 (auto) | 5,532 | 7.5 |

Linear: 0.626 slots per token plus 406. The engine's own startup lines on this box (the s10 start; the 12-GiB starts read 28.0-28.6 and 15.4-16.0 GiB across the campaign): `28.4 GiB available, 12.0 GiB of it still to be taken by the RAM copy, 21.0 GiB of experts read from the files: the file cache can keep the ones that come back`, re-checked after the copy as `15.6 GiB available`, and `RAM budget 12.00 GiB: 9021 of the 19.18 GiB of experts the GPU cache does not hold, by profile rank; the rest are read from the files`. The GPU caches hold 5,573 slots on CUDA0 (7.39 GiB) and 4,586 on CUDA1 (6.45 GiB), and the lend draws from both (at 4096: 1,526 + 1,440). So the file tier at decode is 19.18 - 12.0 = 7.2 GiB; the `21.0 GiB` figure is every expert outside the RAM copy, including the two GPU caches (19.18 + 13.84 - 12.0). During a prompt the lent slots are streamed from the files again, so the set a long prompt reads is 7.2 GiB + lend(chunk) + ~0.35 GiB of PLE pages: about 15.1 GiB at 8192, 13.3 at 6144, 11.6 at 4096. The page cache that was actually available to hold it, measured as the cgroup's file total at steady state, was 14.5-14.6 GiB at budget 12 (MemAvailable after the copy reads ~1 GiB higher, 15.4-16.0). 8192 does not fit, and the reads say so; 6144 fits by about 1.2 GiB and reads 2-3x the bytes of 4096 at the same TTFT, which is the edge; 4096 fits by about 3 GiB.

### Where the choice is made (ed4773eb)

- `src/program/generate.cpp:6669-6748`: `plan_lend` / `biggest_chunk` bisect the chunk against `k + 128 <= xcache.slots() && k * 100 <= kAutoLendPct * xcache.slots()` (`:6733`), where `k = lend_slots(chunk)`. The test is VRAM slots only. The layer-split path has its own copies of the same test per stage (`:4024`, `:4066`, `:7350`, against `parts[i].cnt` / `p.cache->slots()`); this box is a split, so one of those is the test that ran here, and they are VRAM-only too.
- `src/program/generate.cpp:6631-6660`: the #2090 clamp drops the gfx12 ceiling from 16384 to 8192 when `avail_mib < pin_mib + extra_mib + 6144`. It is the only RAM test in the scan; it runs for gfx12 only and stops at 8192.
- `src/core/expert_source.cpp:512-537` and `include/strata/platform/memory.hpp:54`: `file_cache_keeps(avail, arena, read, 1 GiB, 0.05, 1.5 GiB)` decides buffered vs unbuffered reads at load. With `hot_fraction` 0.05 it asks whether a twentieth of the file tier fits; that is the decode question (#1194), not the prompt one, where the lent slots are read once per chunk, in full.
- `src/core/expert_source.cpp:3241-3246`: the split-lend line prints `0 keep their experts in RAM too (... not enough RAM for them)`; the #1389 WARNING at `:3227-3236` is inside the single-GPU `lend` branch (`lend` at `:2746` requires `additional_gpu_pairs.empty()`, and this box has 4,586 of them), so on a split nothing warns that every lent slot will be re-read from the files.

### Proposed change

In the auto scan, at every backend and every step, keep only a chunk whose prompt-time file set fits the RAM that will be available after the RAM copy is built, with a margin:

```
file_set(chunk) = (experts the GPU caches do not hold - RAM budget) + lend_bytes(chunk)
fits(chunk)     = file_set(chunk) + margin <= RAM available for the page cache after the copy
```

Both terms already exist as log lines: the first from `N of the M GiB of experts the GPU cache does not hold` (M - budget; not the `21.0 GiB ... read from the files` line, which counts the GPU caches too), the second from `lend sizing`. On this box, against the 15.4-16.0 GiB MemAvailable after the copy, a margin of 1-2 GiB picks 6144 (13.3 + 2 = 15.3 is still under 15.4) and 4096 needs a margin of about 2.5 GiB; against the ~14.5 GiB the page cache actually held, 2 GiB picks 4096 (13.3 + 2 = 15.3 > 14.5). The measurements say the margin should be the larger one: 6144 fits by the arithmetic but reads 2-3x the bytes of 4096 at the same TTFT and costs 16 s more on the cold first fill, so "largest chunk that fits" should be "largest chunk that fits with 2-3 GiB to spare", or 2 GiB against the page cache the box can actually hold after the copy (MemAvailable overstates it here by ~1 GiB). 8192 would be left to an explicit `--prefill 8192`. On a box where the 8192 set fits with that margin, nothing changes. A second, smaller change: print the #1389 warning on the split path too when `stage_lent < offered`.

### Not tested

One GPU; other quantizations; prompts over 32K; Windows; `--resident-experts` (the full RAM copy). `--ple-io mmap` is used from the 2048-mmap row on because `--ple-io direct` tripped the #29 watchdog on this box (reported separately on #2797); the 2048 direct and 2048 mmap rows differ by under 2 s, so the PLE mode does not move the chunk result.

Logs for every cell (engine log, argv, per-request JSON, diskstats and `/proc` stamps) are available on request.


---
Filed by jds1-ai. Measurements and the /proc instrumentation run by an agent (Fizz, Claude model); source reading, pre-registered scoring and this write-up by a second agent (Fable, Claude model); reviewed by the account owner before filing.