Pull requests / #3086
#3086 expert source: batch-prefetch a layer's CPU experts for a plain mapped experts.bin (STRATA_MMAP_PREFETCH, #3065)
open · @As7656431 · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Title Issue: #3065 (the third of the three changes described there; independent of #3072) ## Summary `FileExpertSource::prefetch` returns at once unless `staged()` (the GGUF in place, an unbuffered file tier or Linux io prefetch). In the low-RAM mode with a pack's `experts.bin` read through the mapping (`--mmap-experts`, no RAM budget), each expert the decode's CPU pool computes is faulted in page by page. With `STRATA_MMAP_PREFETCH` the layer's CPU experts go to the OS as one batch before the pool reads them. Opt-in: with the variable unset nothing changes. ## What changed - `src/core/expert_source.cpp`: `prefetch_mapped` (one `PrefetchVirtualMemory` call per layer on Windows, `madvise(MADV_WILLNEED)` elsewhere; experts in the RAM copy are skipped) and `prefetch_async` (a helper thread; the newest request replaces one that has not started). `prefetch()` uses them when the source is not staged and the variable is set; the destructor stops the thread. - `include/strata/core/expert_source.hpp`: the members for the above. - `STRATA_MMAP_PREFETCH=1` issues the batch on the calling (dispatch) thread; `=2` hands it to the helper thread and returns at once (`PrefetchVirtualMemory` costs about 0.2 ms a call even when every page is resident). `=2` is what I measured. ## Extra Notes - **Same computation.** It only asks the OS for pages ahead of time; the bytes the pool reads are the same. `STRATA_DUMP_FIRST_LOGITS` after an 8K document and a 300-token prompt, two runs each: every run with the variable set was bit-identical to a run without it. Against the 0.1.45 release binary, with the variable unset the code path is the release's; the logits differed only as much as the release differs from itself between runs (#3065), and the first choice was the same in every run. - **Speed**, same PC and settings as #3063 (RTX 5070 12 GB, 31.8 GiB RAM, Windows 10, Flash-Next IQ2_XS, `--mmap-experts`, 0.1.45 server), this branch with the variable unset against set, four interleaved rounds (each first twice), a short question, another, a 20K-token English document, a short question: | | unset | `STRATA_MMAP_PREFETCH=2` | rounds faster | |---|---|---|---| | first request after a start | 35.4 tok/s | 47.0 tok/s (+33%) | 4 of 4 | | warm short chat | 65.2 tok/s | 71.7 tok/s (+10%) | 4 of 4 | | reading the 20K document | 187.5 s | 183.6 s (unchanged) | 2 of 4 | | decode right after the document | 15.2 tok/s | 22.9 tok/s (+50%) | 4 of 4 | | a short question after the document | 35.5 tok/s | 47.7 tok/s (+34%) | 4 of 4 | The gain comes when the mapped pages are not in RAM: after a start and after a long prompt has pushed the cached experts out. On top of #3072 it is smaller (in an earlier run on 0.1.44.1 with both: +22% first request, +11% question after the document, +7% warm chat), since that PR keeps the cached experts in RAM. - Built and tested with CUDA on Windows only. I have not built or tested the non-Windows branch (`madvise`) on Linux or macOS, nor HIP or SYCL.
Sur le site
Liens install, modèles, releases.