Pull requests / #2243

#2243 drop-file-cache: --drop-file-cache returns the model files' page cache after the startup reads (opt-in)

open · @G0-0000 · 0 comentários · No GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindowsLinux

Descrição

## What it changes

New opt-in engine flag `--drop-file-cache` (Linux): after the startup reads complete, and only when a
self-check confirms that no expert is read from the files, the engine issues madvise(MADV_DONTNEED)
followed by posix_fadvise(POSIX_FADV_DONTNEED) over the model files' mapping, which returns the page
cache to the kernel (both calls are needed; either alone does not release the pages).

## What it leaves alone

With the flag absent, no syscall is issued and nothing changes byte for byte.  The self-check refuses
to run the drop whenever any expert is read from the files (mapped-only layouts, partial complements).

## Why

The engine maps the model GGUFs MAP_SHARED and reads the experts through them once; the kernel keeps
those pages for the life of the process, so a 24/7 server holds a full extra copy of the expert data in
the page cache (measured 55.4 GiB on the host below) even though every expert is already in VRAM or in
the page-locked RAM copy - the reads that actually need the cache are only the lend-region refills
(~4 GiB).  Related discussions: #1194, #1682, #1571 (the Windows working-set analogue).

## How it was measured

Same host as the NUMA placement (Supermicro X10DRC, V100-SXM2-32GB, Qwen3.8-Flash-Next UD-IQ4_XS,
125 GB RAM):

- Cached dropped by 50-57 GiB (varies with the baseline page-cache state), in 6.7-8.2 s, once per start;
- node1 free memory +~20 GiB;
- decode: interleaved A/B, 46.6 -> 46.7 tok/s (no regression); the first large prompt re-reads the lent
  region (~1.2-4 GiB) from the drive once (+5-8% on that request), and the pages return through the
  file cache afterwards.

## How it was tested

GPU-free: a synthetic two-shard unit test (mapped pages 3452 -> 0, the advised byte count, and a
byte-identical re-read after the drop) plus the full non-CUDA ctest subset; on-GPU: two controlled
stop-the-service windows (reclaim amount, refill placement, interleave A/B) and a production run.

## Note

A local extension binds the refill reads to the cold NUMA node (so the pages that come back land away
from the GPU's node); it depends on the --numa-hot-node feature and can be sent as a follow-up if this
is of interest.

No site

Links install, modelos, releases.