#2962 conversation cache: parked dialogs give RAM back when available RAM falls below --conversation-cache-min-free-mib

Descripción

`--conversation-cache-min-free-mib` (default 2560) is checked when a conversation is parked, saved or restored. Once parked, a conversation stays in RAM until a later park or restore evicts it. That holds even when the host runs short in between, because another program opened or the engine's own memory grew. This PR makes the floor hold after a park as well: below it, the cache gives RAM back.

### What it does

- **When.** Only with the conversation cache on. It runs before every request, every `STRATA_CONV_PRESSURE_MS` (2000 ms; 0 = off) while the engine is idle, and between batch windows. The idle wait for the next stdin line wakes in 2 s slices for it.
- **The reading.** The one the park gate uses (`sample_host_memory()`: on Windows the smaller of available physical memory and commit room; on Linux MemAvailable, the cgroup limits and `--memory-limit-mib`). Above the floor that one reading is all it does. An unknown reading releases nothing, as the park gate fails closed.
- **What goes, oldest first, until the floor + `STRATA_CONV_PRESSURE_HYST_MIB` (1024) is available:**
  1. the retained K/V (an optimization only);
  2. the parked conversations, least recently active first, through the cache's own `evict_oldest()`. Pinned shared prefixes and a guarded donor stay. A client that comes back has its prompt read again;
  3. the idle `--batch` slots' host checkpoints. Their K/V and ids in VRAM stay; a partial read keeps its own.
- **After each step** it re-reads. glibc gets `malloc_trim` first: a freed 16 MiB mmap chunk raises glibc's mmap threshold, so later segments come from the heap and stay in the process without it. On Windows the CRT heap hands blocks this large back on free.
- **One stderr line per event.** For example `conversation cache: RAM below the floor: released 2 dialogs, 2180 MiB (dropped) in 131 ms; idle: 9761 -> 11901 MiB available, floor 10240 MiB + 1024; parked=1`. An event with nothing left to release is said once, until a request ran or the held bytes changed.
- **The code.** The policy is a header (`include/strata/core/conversation_pressure.hpp`), so it is tested without a GPU. generate.cpp only wires it in.

### Default behaviour

The cache is off by default (`--conversation-cache-mib 0`), and then nothing runs: the stdin wait is the old `next_line`. With the cache on and RAM above the floor, the answers are the same; only the 2 s reading is added. The floor's default (2560) is unchanged.

### Measured on this branch

RTX 5090 (Windows 11, CUDA 13.3, driver 617.14), 9950X3D, 128 GB. upstream/main 477ac13c and this branch, the same release configure. The NVFP4 GPTQ pack (`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata`, upstream setup's `--family nvfp4 --model NVFP4`).

**Identity.** `--exact --cache 3000`, the p2k and long_32k prompts, first-token logits (sha256) and 32 greedy tokens:

| run | p2k logits | long_32k logits | greedy tokens |
| --- | --- | --- | --- |
| upstream/main | `1e9c612c...` | `f5b2872c...` | 4 / 20 |
| this branch | `1e9c612c...` identical | `f5b2872c...` identical | identical |
| this branch, `--conversation-cache-mib 8192 --conversation-cache-slots 4` (p2k parked when long_32k came, no pressure) | `1e9c612c...` identical | `f5b2872c...` identical | identical |

**Under pressure** (79 s in all). The server ran setup's config for this PC (`--expert-cache auto --spec 4 --spec-min-p 0.5 --max-context 262144 --kv int8 --kv-resident 32768`, images off) plus `--conversation-cache-mib 8192 --conversation-cache-slots 4 --conversation-cache-min-free-mib 10240`.
- Three dialogs with a shared 3K system prompt were sent one turn each: B 32K, A 48K, E 64K. A short side dialog Q came last, so E parked too. Parked: 3, 3,974 MB. 42,057 MiB available, engine private bytes 107,297 MiB.
- 6 s idle: no release line.
- A ballast program took 32,512 MiB in 4.1 s, down to 9,745 MiB available, under the floor.
- 1.5 s later, at the next idle check, the engine said: `conversation cache: RAM below the floor: released 2 dialogs, 2180 MiB (dropped) in 131 ms; idle: 9761 -> 11901 MiB available, floor 10240 MiB + 1024; parked=1`.
  - The oldest two went: B (962 MB) and A (1,325 MB). Together they reached the target, so E stayed.
  - The engine's private bytes fell 107,297 -> 105,148 MiB (-2,149) at once.
- The ballast stayed 10 s in all: one line only, no repeat.
- Follow-ups after the ballast was gone:
  - B (released): `32092 tokens = 2998 reused + 29094 read`, TTFT 5.3 s; its own codename in the answer.
  - E (still parked): `64093 tokens = 64047 reused + 46 read`, restored in 133 ms, TTFT 0.54 s; its own codename.
- 4 s idle after: no release line.

### Measured earlier on our fork

The same change on our fork's 0.1.44 code, which also has a disk tier (sergqwer/strata-nvfp4), RTX 5090, the same pack. Config: cache 8192 MiB x 4, floor 10240. Four dialogs of 32K-100K were parked (5.6 GB), then a ballast took available RAM down to 8.6 GB. 2.0 s later the engine said `released 3 dialogs, 3338 MiB (dropped) in 104 ms; idle: 8642 -> 11937 MiB available`. The newest one stayed parked and was restored in 0.19 s; a released one was read again. The engine's private bytes went down 3283 MiB. Nothing was released without pressure. With `--batch 2` and both dialogs cached in slots, the parked images went first and then the slots' turn checkpoints (2180 MiB together); the slots still reused 32K/48K tokens afterwards.

### Tests

`conversation_pressure_test` (new, CPU, 63 checks; MSVC, MinGW `g++ -Wall -Wextra`, Linux g++ 13.3 / glibc 2.39). The checks, with fake readings:
- at the floor and above it, nothing is released and one reading is taken;
- inside the hysteresis band, nothing;
- below the floor the oldest go until floor + hysteresis, and the newest stay;
- settling alone can be enough;
- the retained K/V goes first;
- pinned prefixes stay, and the event is then exhausted;
- an unknown reading releases nothing, and a reading lost midway stops the release;
- at most `bound` units go.

One real check: 256 MiB parked and released leaves the process (Windows private bytes 257 -> 0 MiB; Linux RSS 299 -> 3 MiB, twice, the second time after the mmap threshold rose). The standalone conversation tests (`conversation_cache_test`, `conversation_file_test`, `conversation_memory_test`, `conversation_split_failure_test`, `session_save_reclaim_test`) pass.

### How to reproduce

- Install with upstream setup: `--family nvfp4 --model NVFP4`.
- Add `--conversation-cache-mib 8192 --conversation-cache-slots 4 --conversation-cache-min-free-mib 10240` to the config's args.
- Park a few long dialogs, then take RAM away with any program until less than the floor is available. Within 2 s the line above appears.

### Not covered

- The live session's own checkpoint chain (`--prompt-cache`, ~118 MB each): it is the active conversation, not a parked one.
- `STRATA_CKPT_POOL`'s at most two kept checkpoints.
- The SYCL copy of generate.cpp: unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio