#3115 0.1.45: segment_capacity's held/8 bound shrinks pool segments below the mmap threshold; evicted memory lingers in the glibc arena and the RAM admission loop drains the whole conversation cache

说明

## Environment

- Strata 0.1.45 (engine self-report; built from main), Ubuntu 24.04, glibc 2.39
- 4x RTX 4090, 566 GB RAM, 1M context, `--batch 4 --batch-groups auto`, `--kv k8v4 --kv-resident 524288`
- `--conversation-cache-mib 393216 --conversation-cache-slots 192 --conversation-cache-min-free-mib 65536 --conversation-cache-min-tokens 2000`
- Workload: long-context multi-turn agent traffic, prompts 200K-680K tokens, a few concurrent conversations

## Observed on 0.1.45 (one day of traffic)

- The conversation cache drained from **125 entries / 383 GiB to 20 entries / 38.8 GiB**.
- Admission evictions: **1529 conversations over 237 loops**; the largest single loop evicted **155 conversations to admit a snapshot needing only 2957 MiB** against the 65536 MiB floor.
- Engine RSS grew to **444 GiB (391 GiB anonymous)**; `/proc/<pid>/maps` shows **`[heap]` = 341.3 GiB in a single brk segment** - the evicted conversations' pages linger in the glibc arena instead of returning to the kernel.
- `MemAvailable` stays pinned at 76-80 GiB, so the admission line (`need + 64 GiB floor`) is not reached, the loop keeps evicting (`while (!admit() && evict_oldest())`), and the pool gets drained.
- Downstream effect: follow-up turns of long conversations miss the cache and re-read in full (observed reuse 8.3% / 6.7% / 19.5% on 200K+ prompts, ~90-126 s of prefill each).

For contrast, on **0.1.44.1** with the same box, workload and `MALLOC_MMAP_THRESHOLD_=16777216`, admission was well-behaved: a snapshot admission evicted **1-3 conversations** (the freed pages returned to the kernel immediately and `MemAvailable` crossed the line after the first evictions).

## Root cause

0.1.45's `ConversationBuffer::segment_capacity(count, previous, growing, held)` bounds the append reserve by `held/8` (at least 64 KiB) - a welcome memory saving (~40% per conversation by our measurement), but it means **most segments of a parked conversation are smaller than 16 MiB** (while `held < 128 MiB` every segment is, and small conversations stay below entirely).

Segments below glibc's mmap threshold are served from the arena; `free()` puts them on arena free lists and the pages are **not returned to the kernel**, so `MemAvailable` does not move. The admission loop samples `/proc/meminfo` after every eviction, sees "still not enough room", and keeps evicting - past the point where hundreds of GiB have actually been freed inside the process.

On 0.1.44 the `previous*2` growth reached the 16 MiB cap quickly, so setting `MALLOC_MMAP_THRESHOLD_=16 MiB` made every segment its own mmap and the loop terminated after 1-3 evictions. On 0.1.45 that no longer covers the pool.

## Workaround we deploy (works, but environment-level)

`MALLOC_MMAP_THRESHOLD_=1048576` (1 MiB): every segment >= 1 MiB becomes its own mmap and eviction returns it at once. With `vm.max_map_count` raised (1M here) the VMA budget holds (~100K VMAs at a full 384 GiB pool). Sub-1-MiB segments still linger, and we would rather not depend on libc tuning for correctness.

## Suggested fixes

1. **Minimal and robust**: call `malloc_trim(0)` during the admission eviction loop (e.g. every 32 evictions and once after the loop). On glibc this returns arena free pages to the kernel regardless of thresholds, keeps the meminfo-based admission honest, and is essentially free elsewhere. This alone would have prevented the pool drain.
2. Alternatively: size segments so they stay at/above a documented mmap threshold (and document the variable), or
3. Longer term: allocate `ConversationBuffer` segments directly with `mmap(MAP_ANONYMOUS)` / release with `munmap` - the code comment in `conversation_cache.hpp` already describes each segment as "its own allocation"; making that literal would decouple the admission loop from libc policy entirely.

Happy to provide smaps dumps or test candidate patches on our 4-GPU box. Thanks for 0.1.45 - the piped read and split overlap are a big win for multi-turn long-context traffic.