改动 / #3072
#3072 prompt path: STRATA_PROMPT_DIRECT=1 on Windows, also in the low-RAM mode without a RAM budget (#3065)
open · @As7656431 · 0 评论 · 去 GitHub 看
BenchmarksAMD / HIPNVIDIA / CUDAWindowsLinux
说明
## Title Issue: #3065 (the first of the three changes described there) ## Summary On a Windows PC whose RAM cannot hold the experts (`--mmap-experts`), the stager copies each prompt chunk's experts out of the mapped `experts.bin`: millions of page faults that read the drive slowly and push the decode's cached experts out of RAM. `STRATA_PROMPT_DIRECT=1` already reads the prompt path's blobs unbuffered on Linux, in the RAM-budget branch only. This PR makes it work on Windows, and also in the low-RAM mode without a RAM budget. Opt-in: with the variable unset nothing changes. ## What changed - `src/core/expert_source.cpp`: a Windows branch in `open_prompt_direct` (experts.bin opened a second time with `FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED`); `read_direct` uses these handles when the file tier itself is not unbuffered. `copy_blobs` is unchanged: it already reads through `pdirect_`. - `include/strata/core/expert_source.hpp`: `ExpertSource::prompt_reads()` (default false; true on Windows for experts.bin with those handles open). - `src/prefill/prefill.cpp`: with `prompt_reads()` the stager hands unpinned blobs to `copy_blobs` (8 neighbours per request, ring of 2 x batch x threads). `transient()` is not touched, so the decode and the prompt CPU share keep their behaviour. - `src/program/generate.cpp`: on Windows, `STRATA_PROMPT_DIRECT=1` is also honoured in the mmap mode without `--resident-budget-gib`. ## Extra Notes - **Default path unchanged / same computation.** `STRATA_DUMP_FIRST_LOGITS` after an 8K document and a 300-token prompt, two runs each, against the 0.1.45 release binary: with the variable unset and with it set, every run of this build was bit-identical to one of the release's runs (the release's own two runs of the 8K document differ, KL 1.2e-3, as noted in #3065). - **Speed, same PC and settings as #3063 (RTX 5070 12 GB, 31.8 GiB RAM, Windows 10, Flash-Next IQ2_XS, `--mmap-experts`), `tools/strata_bench.py`, medians of 3; the 0.1.45 numbers are from #3063, measured the same day:** | | 0.1.45 | this PR, `STRATA_PROMPT_DIRECT=1` | |---|---|---| | 4K prompt | 58.3 tok/s, first token 69.0 s | 160.6 tok/s, first token 25.0 s | | 16K prompt | 127.5 tok/s, first token 126.0 s | 310.8 tok/s, first token 51.7 s | | agent follow-up turn, first token | 48.2 s | 25.0 s | | agent decode | 6.4 tok/s | 51.8 tok/s | The agent decode gains most because the prompt no longer pushes the decode's cached experts out of RAM. From the command line, an 8K document read in 73.7 / 75.7 s with the release and 28.3 / 28.4 s with this PR. - **Linux is unchanged**, also with the variable set: `prompt_reads()` and the new branch in `generate.cpp` are Windows only. - On Windows, a RAM budget together with `STRATA_PROMPT_DIRECT=1` (which `open_prompt_direct` used to refuse there) now opens the handles too; I have not tested that combination. - NTFS serving unbuffered reads of a mapped file one at a time (the note in `read_direct`) still applies; on this drive (about 1.4-1.6 GB/s) the reads run near the drive's speed, which is far more than the page faults got. Not tested on a fast NVMe, where the default path is already much faster (#2791). - Built and tested with CUDA on Windows only. The new code is Windows file I/O, not GPU code, but the HIP build on Windows compiles it too and I have not built HIP or SYCL.
本站相关内容
相关页面的快捷入口。