贡献 / #2243
#2243 drop-file-cache: --drop-file-cache returns the model files' page cache after the startup reads (opt-in)
open · @G0-0000 · 0 评论 · 去 GitHub 看
BenchmarksNVIDIA / CUDAModels & quantsWindowsLinux
说明
## What it changes New opt-in engine flag `--drop-file-cache` (Linux): after the startup reads complete, and only when a self-check confirms that no expert is read from the files, the engine issues madvise(MADV_DONTNEED) followed by posix_fadvise(POSIX_FADV_DONTNEED) over the model files' mapping, which returns the page cache to the kernel (both calls are needed; either alone does not release the pages). ## What it leaves alone With the flag absent, no syscall is issued and nothing changes byte for byte. The self-check refuses to run the drop whenever any expert is read from the files (mapped-only layouts, partial complements). ## Why The engine maps the model GGUFs MAP_SHARED and reads the experts through them once; the kernel keeps those pages for the life of the process, so a 24/7 server holds a full extra copy of the expert data in the page cache (measured 55.4 GiB on the host below) even though every expert is already in VRAM or in the page-locked RAM copy - the reads that actually need the cache are only the lend-region refills (~4 GiB). Related discussions: #1194, #1682, #1571 (the Windows working-set analogue). ## How it was measured Same host as the NUMA placement (Supermicro X10DRC, V100-SXM2-32GB, Qwen3.8-Flash-Next UD-IQ4_XS, 125 GB RAM): - Cached dropped by 50-57 GiB (varies with the baseline page-cache state), in 6.7-8.2 s, once per start; - node1 free memory +~20 GiB; - decode: interleaved A/B, 46.6 -> 46.7 tok/s (no regression); the first large prompt re-reads the lent region (~1.2-4 GiB) from the drive once (+5-8% on that request), and the pages return through the file cache afterwards. ## How it was tested GPU-free: a synthetic two-shard unit test (mapped pages 3452 -> 0, the advised byte count, and a byte-identical re-read after the drop) plus the full non-CUDA ctest subset; on-GPU: two controlled stop-the-service windows (reclaim amount, refill placement, interleave A/B) and a production run. ## Note A local extension binds the refill reads to the cold NUMA node (so the pages that come back land away from the GPU's node); it depends on the --numa-hot-node feature and can be sent as a follow-up if this is of interest.
本站相关内容
相关页面的快捷入口。