#5504 [Engine] Credit clean active_file cache in the conversation memory guard

closed · @giphoo · 0 comments · View on GitHub

NVIDIA / CUDAWindowsLinux

Description

## Problem and result

The conversation memory guard refused session reads on an mmap-heavy engine because `reclaimable_cache()` only credited `inactive_file`, giving 0 reclaimable bytes even when 17+ GiB of clean file cache sat on the active LRU list.

After this change, both `inactive_file` and `active_file` are credited (less dirty and writeback, capped at the cgroup's current usage).

## Related discussion

none

## Approach

The current comment says `active_file (it holds mlocked pages)` - but on cgroup v2, mlocked pages are counted in `unevictable`, not `active_file`. Clean `active_file` pages are regular file cache on the active LRU list and are reclaimable by the kernel.

An mmap-heavy engine (model weights read through mappings) naturally keeps its file cache on the active list. Measured on a production system:

```
inactive_file: 0
active_file:   17.5 GiB
file_dirty:    0
unevictable:   0
```

The guard read 0 credit and refused a 50 MiB session file read with 28 GiB actually free.

This PR credits `inactive_file + active_file`, still less dirty and writeback, still capped at the cgroup's charged usage, and still does not credit shmem (the KV pool's pinned complements).

## Validation

**Passed (reference implementation, Linux, cgroup v2, mmap-heavy workload):**
- Session read admission succeeds when active_file holds the file cache
- 51-request conversation rotation with 0 OOM after the change

**Failed:** none observed

**Not run:**
- cgroup v1 (`total_active_file` path is added but untested)
- Windows (no cgroup memory.stat)
- CUDA, SYCL, Metal (host-side only)

## Limits

- Crediting `active_file` assumes the kernel can reclaim those pages under pressure. On a system with genuine memory shortage, reclaiming active file cache causes I/O to re-read the pages later, trading latency for admission.
- cgroup v1 path added but not tested.

Related on this site

Quick links to related pages.