#3065 [Feature]: Windows low-RAM mode (--mmap-experts): let STRATA_PROMPT_DIRECT work there - prompt reads are page-fault bound (opt-in patches and measurements)

本文

**Goal:** on a Windows PC whose RAM cannot hold the experts (low-RAM mode, `--mmap-experts`), read prompts without millions of page faults, and keep the decode's cached experts in RAM afterwards. I have three opt-in changes for this, measured on one PC, and would like to know whether you would take them as pull requests, or prefer another approach.

**The PC:** RTX 5070 12 GB, i5-14600KF, 31.8 GiB DDR4-3600, Windows 10 22H2, model files on an aigo P2000 NVMe (about 1.4-1.6 GB/s). Original Flash-Next IQ2_XS (35.5 GB of experts), `--mmap-experts`, 32K context, `--kv int8`, MTP. Results-only report for 0.1.45 on this PC: #3063.

## What happens

- A 4K prompt reads at 58 tok/s and a 16K prompt at 128 tok/s (0.1.45, `tools/strata_bench.py`); agent follow-up turns wait 48 s for the first token and then decode at 6.4 tok/s.
- With Windows performance counters during a 20K-token prompt (0.1.39), the drive read only 0.2-0.5 GB/s while about 300 MB of RAM stayed available: the stager's copies out of the mapped `experts.bin` are page faults, not drive-bound reads. They also push the decode's cached experts out of RAM, so the answer after a long prompt decodes at 13-19 tok/s, against about 70 tok/s for a warm chat.
- #2004 (RTX 4060 8 GB, 32 GB, Windows 10, IQ2_XS low-RAM) describes the same thing: about 31-36 tok/s on a 9.7K prompt, one stall "with heavy paging of expert file pages". #1520 attacks it from another side (releasing the copied pages).

## Why the existing switches do not cover it

- `STRATA_PROMPT_DIRECT=1` is read only in the RAM-budget branch of `generate.cpp`, and `FileExpertSource::open_prompt_direct` is Linux only ("NTFS serves unbuffered reads of a mapped file one at a time").
- `FileExpertSource::prefetch` returns at once unless `staged()`, so a plain mapped `experts.bin` gets no batch prefetch in decode; each CPU expert faults in page by page.
- A RAM budget (`--resident-budget-gib 8` / `14`, 0.1.44.1) reads a 20K prompt in 80 s instead of about 170 s, but decodes at about 14 tok/s on this PC, so it is not an alternative here.

## The changes (on 0.1.44.1; every one off by default)

1. **`STRATA_PROMPT_DIRECT=1` on Windows and without a RAM budget.** A Windows branch in `open_prompt_direct` (a second handle on `experts.bin` with `FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED`; `read_direct` uses these handles when the tier itself is not unbuffered), the switch honoured in the mmap mode without a budget, and a new `ExpertSource::prompt_reads()` so the stager hands unpinned blobs to `copy_blobs` (8 per batch) without touching `transient()`: the decode and the prompt CPU share behave exactly as before.
2. **`STRATA_PREFILL_DIRECT_RAM=3`.** In that `copy_blobs` path, a blob whose pages are all in the process working set (`QueryWorkingSetEx`) is copied from the mapping; the rest are read through the file cache as merged runs of neighbouring blobs (the standby list still holds many of them). On this PC about 70-77% of the prompt's expert bytes came from the working set and none had to be read unbuffered.
3. **`STRATA_MMAP_PREFETCH=2`.** For a plain mapped `experts.bin`, the layer's CPU experts go to `PrefetchVirtualMemory` as one batch on a helper thread (the call costs about 0.2 ms even when every page is resident, so the decode thread does not wait for it).

Together with two switches that already exist: `STRATA_PREFILL_STREAM_MIN=8193` (stream only the routed experts for chunks up to 8,192 tokens) and `STRATA_WATCHDOG_S=180`. About 300 lines in `expert_source.cpp/hpp`, `prefill.cpp` and `generate.cpp`. Built and tested only with CUDA on Windows; the new code is Windows file I/O, not GPU code, but I have not built HIP or SYCL.

## Measurements (same PC, same day, the same settings apart from the switches)

`tools/strata_bench.py` (the 0.1.45 tool as the client), 0.1.44.1 against 0.1.44.1 with the switches, medians of 3:

| | 0.1.44.1 | with the switches |
|---|---|---|
| 4K prompt | 70.7 tok/s, first token 56.9 s | 978.5 tok/s, first token 4.1 s |
| 16K prompt | 95.7 tok/s, first token 167.9 s | 1455.9 tok/s, first token 11.1 s |
| agent follow-up turn, first token | 49.1 s | 5.7 s |
| agent decode | 10.9 tok/s | 66.1 tok/s |
| short-chat decode | 63.9 tok/s | 61.8 tok/s |

The tool's prompts are generated word lists, which probably route to fewer experts than real text and so flatter `STRATA_PREFILL_STREAM_MIN`. With a real 20K-token English document (four versions in one run, four rounds, a different version first in each round; averages of the four rounds):

| | 0.1.39 | 0.1.44.1 | 0.1.44.1 with the switches |
|---|---|---|---|
| read the 20K document | 173.6 s | 162.5 s | 33.1 s |
| decode right after it | 13.3 tok/s | 17.4 tok/s | 42.0 tok/s |
| a short question after it | 26.0 tok/s | 40.1 tok/s | 65.0 tok/s |
| warm short chat | 54.2 tok/s | 71.3 tok/s | 74.4 tok/s |
| first request after a start | 31.6 tok/s | 41.2 tok/s | 48.1 tok/s |

`STRATA_MMAP_PREFETCH=2` alone, four interleaved rounds: first request after a start +22% (4 of 4 rounds faster), a short question after the long document +11% (3 of 4), warm chat +7% (4 of 4), prompt reading unchanged.

## Correctness

The switches change where the bytes come from, not what is computed. Checked with `STRATA_DUMP_FIRST_LOGITS` (the logits after the prompt) on an 8K document and a 300-token prompt:

- 8K document: all six runs with switches 1 and 2 (alone and with the existing switches) were bit-identical to one of the two official runs.
- 300-token prompt: the four runs with switches 1 and 2 were bit-identical to the official runs. One of the runs with all switches gave an output no official run gave, but so did the official binary with only the existing `STRATA_PREFILL_STREAM_MIN` / `STRATA_WATCHDOG_S` switches, and the official binary alone gave three different outputs in seven runs. The extra output is within that spread (KL up to 1.8e-4 against the official outputs, which differ from each other by up to 2.1e-4); the first choice was the same in every run.
- `STRATA_MMAP_PREFETCH` only asks the OS for pages ahead of time, so the bytes the pool reads are the same.

So one thing I noticed on the way: the official binary itself is not bit-reproducible between runs on this PC (two runs of the 8K document differ by a KL of 2.4e-3).

## Not tested

Linux; other GPUs; a fast NVMe (your note in `read_direct` measured unbuffered reads of a mapped file at 3.4 GB/s on a PCIe 5 drive; on this drive they still beat the page faults, but on a fast drive the official path is already much faster, see #2791); a RAM budget together with the switches; GGUF-in-place packs; HIP and SYCL builds; output quality beyond the logits check (no change is expected, since the computation is the same).

If this is welcome, I would send three separate PRs (1, 2, 3 above) on top of current main, each off by default with the default path unchanged.