改动 / #3072

#3072 prompt path: STRATA_PROMPT_DIRECT=1 on Windows, also in the low-RAM mode without a RAM budget (#3065)

open · @As7656431 · 0 评论 · 去 GitHub 看

BenchmarksAMD / HIPNVIDIA / CUDAWindowsLinux

说明

## Title
Issue: #3065 (the first of the three changes described there)

## Summary
On a Windows PC whose RAM cannot hold the experts (`--mmap-experts`), the stager copies each prompt chunk's experts out of the mapped `experts.bin`: millions of page faults that read the drive slowly and push the decode's cached experts out of RAM. `STRATA_PROMPT_DIRECT=1` already reads the prompt path's blobs unbuffered on Linux, in the RAM-budget branch only. This PR makes it work on Windows, and also in the low-RAM mode without a RAM budget. Opt-in: with the variable unset nothing changes.

## What changed
- `src/core/expert_source.cpp`: a Windows branch in `open_prompt_direct` (experts.bin opened a second time with `FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED`); `read_direct` uses these handles when the file tier itself is not unbuffered. `copy_blobs` is unchanged: it already reads through `pdirect_`.
- `include/strata/core/expert_source.hpp`: `ExpertSource::prompt_reads()` (default false; true on Windows for experts.bin with those handles open).
- `src/prefill/prefill.cpp`: with `prompt_reads()` the stager hands unpinned blobs to `copy_blobs` (8 neighbours per request, ring of 2 x batch x threads). `transient()` is not touched, so the decode and the prompt CPU share keep their behaviour.
- `src/program/generate.cpp`: on Windows, `STRATA_PROMPT_DIRECT=1` is also honoured in the mmap mode without `--resident-budget-gib`.

## Extra Notes
- **Default path unchanged / same computation.** `STRATA_DUMP_FIRST_LOGITS` after an 8K document and a 300-token prompt, two runs each, against the 0.1.45 release binary: with the variable unset and with it set, every run of this build was bit-identical to one of the release's runs (the release's own two runs of the 8K document differ, KL 1.2e-3, as noted in #3065).
- **Speed, same PC and settings as #3063 (RTX 5070 12 GB, 31.8 GiB RAM, Windows 10, Flash-Next IQ2_XS, `--mmap-experts`), `tools/strata_bench.py`, medians of 3; the 0.1.45 numbers are from #3063, measured the same day:**

| | 0.1.45 | this PR, `STRATA_PROMPT_DIRECT=1` |
|---|---|---|
| 4K prompt | 58.3 tok/s, first token 69.0 s | 160.6 tok/s, first token 25.0 s |
| 16K prompt | 127.5 tok/s, first token 126.0 s | 310.8 tok/s, first token 51.7 s |
| agent follow-up turn, first token | 48.2 s | 25.0 s |
| agent decode | 6.4 tok/s | 51.8 tok/s |

  The agent decode gains most because the prompt no longer pushes the decode's cached experts out of RAM. From the command line, an 8K document read in 73.7 / 75.7 s with the release and 28.3 / 28.4 s with this PR.
- **Linux is unchanged**, also with the variable set: `prompt_reads()` and the new branch in `generate.cpp` are Windows only.
- On Windows, a RAM budget together with `STRATA_PROMPT_DIRECT=1` (which `open_prompt_direct` used to refuse there) now opens the handles too; I have not tested that combination.
- NTFS serving unbuffered reads of a mapped file one at a time (the note in `read_direct`) still applies; on this drive (about 1.4-1.6 GB/s) the reads run near the drive's speed, which is far more than the page faults got. Not tested on a fast NVMe, where the default path is already much faster (#2791).
- Built and tested with CUDA on Windows only. The new code is Windows file I/O, not GPU code, but the HIP build on Windows compiles it too and I have not built HIP or SYCL.

本站相关内容

相关页面的快捷入口。