#3324 [Benchmark] RTX 4090 with the model in the page cache: experts read in place from the GGUF cost 21-57% prompt, 19-24% decode (file tier of #149)

说明

## Problem and result

This machine keeps the whole model in the page cache (40 GiB of experts, 46 GiB of RAM), so the file tier
of #149 is off: the engine reads experts from a packed `packs/iq3_xxs/experts.bin` and says
`experts via mmap (--mmap-experts; the A/B arm of R2.1)`. The pack also carries `native_experts.txt` with
the offsets inside the GGUF, so switching to the tier is a one-line change. We measured the switch.

Result: **the file tier costs 21% to 57% of prompt reading and 19% to 24% of decode here, and 0 of 2
interleaved pairs favoured it.** The pack stays. This is a boundary-of-feature measurement, not a bug
report: the tier is for machines that cannot hold the file set in memory, and this one can - the plan
line of the engine itself says `file set 31.7 GiB fits the page cache 40.1 GiB -> about 0.0 GB read from
the drive per 32768-token prompt`, so the mechanisms have nothing to overlap.

Two related numbers are in the same folder: `STRATA_IO_PREFETCH=1` on the packed file (the opt-in that
does exist for a pack) makes decode slower in 3 of 3 pairs (median -6.2%), so it stays off; and the drive
speed the prefill plan assumes (`drive 1.50 GB/s (assumed)`) is 2-8x below what this NVMe actually does
(3.6-12 GB/s in six direct-access reads), which does not change this plan but would matter where the file
tier is the bottleneck.

## Related discussion

none (a results-only submission; no engine change is proposed). Follows the engine A/B PR of the same
day for the same machine, where the same configuration and the same instruments are described.

## Approach

`def` is the working configuration; `gguf` is the same configuration with `packs/iq3_xxs/experts.bin`
temporarily renamed away, which makes the engine read the experts in place from the GGUF and turn the tier
on. Nothing else differs; the script puts the pack back. The change is verified from the engine's own
output and from the files it actually mapped:

```text
def : experts via mmap (--mmap-experts; the A/B arm of R2.1)      -> maps packs/iq3_xxs/experts.bin
gguf: experts via mmap (--mmap-experts; the GGUF shards in place, no experts.bin)
      file tier (Linux defaults, the GGUF read in place): … (STRATA_COLD_OVERLAP=4, io prefetch on 8 I/O threads, STRATA_IO_PF_GATE=1)
                                                                  -> maps models/IQ3_XXS/…-00001-of-00002.gguf
```

Two interleaved pairs, a freshly started engine per arm, 3 runs per case, medians; the auto-chunk is
8192 tokens / 96 slots in both arms, so the two sides are comparable token by token.

## Validation

**Passed:**

- the A/B above on commit `e88ee39d` (engine 0.1.45.3, source build, arch `[89]`), Ubuntu 26.04.1,
  NVIDIA 615.71.09, RTX 4090 24 GB, Ryzen 9 7950X, 46 GiB RAM. Observed medians, pack vs GGUF-in-place
  (prompt tok/s | decode tok/s): 4K 1972.7 vs 844.9 | 127.4 vs 97.5; 32K 3309.7 vs 1879.6 | 131.9 vs
  100.8; 128K 3285.5 vs 2567.2 | 127.1 vs 103.5; 194K 3249.9 vs 2553.7 | 131.7 vs 101.3;
  generation-only 297.6 vs 130.5 | 157.4 vs 124.8. TTFT grows 1.3-2.8x. The gap narrows with prompt
  length (x0.43 at 4K, x0.78 at 194K) and never closes; 0 of 2 pairs in favour of the tier on any case.
- draft acceptance did not change (942/1393 vs 983/1329 in the 4K case, inside the run-to-run spread), so
  the same tokens are produced through a worse data path, not a different routing.
- `STRATA_IO_PREFETCH=1` against the same-session baseline, 3 interleaved pairs: prompt 32K x1.003
  (2/3 pairs), 4K x0.914, decode slower in 3/3 pairs (159.1 -> 148.3, 149.6 -> 140.3, 148.6 -> 143.3).
- `dd iflag=direct` on `experts.bin`, six samples: 3.6-12 GB/s on an ADATA LEGEND 960 (raw output in
  `drive-read.txt`).

**Failed:** none observed beyond the numbers above; nothing here is a failing test or a crash.

**Not run:** the case where the tier should actually pay - a cold start with an empty page cache, and a
host with 16-32 GB of RAM where the 31.7 GiB file set does not fit. We do not have such a host: our
second machine (4x Tesla V100-SXM2 32 GB) keeps 98% of the experts resident, so it is not the target
regime either. Other `STRATA_COLD_OVERLAP` / `STRATA_IO_PF_*` values than the defaults, and
`--pcie-frac` variants, were not swept.

## Limits

One quant (IQ3_XXS), one pack, one machine with the model fully cacheable; our own prompt suite (the one
described in the engine A/B PR of the same day), not the repository's suite, so these ratios are not
directly comparable with other community reports - the two arms are identical instruments of the same
suite. The conclusion is about the regime "model fits the page cache", not about the tier in general.