Pull requests / #2253

#2253 serve: the RAM gate releases the heap and stops evicting when it cannot help (rebased #1394)

open · @alanthinker · 0 comentarios · En GitHub

Server & APINVIDIA / CUDAModels & quants

Descripción

## Why

Follow-up to #1347 / #1394, rebased onto current `main` (v0.1.43.1). #1394 is now unmergeable (`mergeable_state: dirty`) because main has since gained the pinned shared prefixes (#879), `sample_host_memory()` with the cgroup limits (#1601) and `--session-save-reclaim` (#1573). This branch carries the same fix over those.

Main's physical-RAM gate evicts parked conversations one at a time and re-reads the host's free RAM after each, on the assumption that dropping a conversation is visible to `/proc/meminfo` at once (the comment in `generate.cpp` still says so: *"an evicted entry is back with the kernel before the next check reads /proc/meminfo; no waiting is needed"*). **That assumption is wrong for the allocation this cache actually makes.**

`ConversationBuffer` is a list of 16 MiB segments (`segment_bytes = 16 * 1024 * 1024`), each one `operator new`. glibc's mmap threshold is *dynamic*: the first generation of segments is mmap'd and `munmap()` returns it, but freeing an mmap'd block larger than the threshold raises the threshold to that size (up to 32 MiB), so every later 16 MiB segment comes from the main heap instead. Freeing one of those only moves the bytes onto the allocator's free list — unless the chunk reaches the top of the heap, RSS, and with it `MemAvailable`, does not move at all.

Reproduced here with a stand-alone program that parks and drops 8 x 16 MiB segments in two generations (same box, same glibc):

```
gen1 parked (8 x 16 MiB)      : RSS 129 MiB
gen1 all dropped              : RSS 129 MiB
gen2 parked (8 x 16 MiB)      : RSS 129 MiB
gen2: evict ONE mid-heap      : RSS 129 MiB   <- the reading does not move
gen2: after malloc_trim(0)    : RSS 113 MiB   <- only now is it visible
```

In the engine the effect is worse than in the toy: measured on this serving box, freeing a 64 MiB snapshot in the middle of the heap moved RSS by **-0.3 MB**, and `malloc_trim(0)` then returned **127.7 MB**. So the loop can evict conversation after conversation while the figure never moves, and in the worst case empty the whole cache and still refuse the snapshot — worse than the refusal it was meant to remove, and it throws away the prompt read that built it (the original report: 270k tokens, 96 s, 32 times).

## What changed

- `conversation_release_freed_memory()` (`malloc_trim(0)` under `__GLIBC__`, a no-op elsewhere) is called before the first re-read and after every eviction, so a dropped conversation shows up in the next reading.
- `ConversationCache::admit_ram()` gives up **before** evicting anything when the missing bytes exceed `evictable_bytes()`, so a shortfall the parked conversations cannot cover no longer empties the cache for nothing; it re-checks the figure after each reading, so losing telemetry ends the loop.
- Adapted to what main has since gained: the reading stays the memory guard's (`sample_host_memory()`: `MemAvailable`, the cgroup limits above the engine, `--memory-limit-mib`) and the refusal line keeps `ram_note()`; `evictable_bytes()` counts only the entries `evict_oldest()` may actually take, i.e. those **not** holding a pinned shared prefix.

## Tests

#1394's cases, which cover what main's current gate does not: a simulated host that returns half, a fifth, or none of what was dropped, another process taking RAM between evictions, a host already under the floor, an empty cache, and no telemetry at all (136 `check()` sites here against the 109 in main's version).

```
$ cmake -B build -DSTRATA_BUILD_CONVERSATION_TESTS=ON && cmake --build build --target conversation_cache_test conversation_memory_test
$ ./build/conversation_cache_test
conversation_cache_test: 4276 checks passed
$ ./build/conversation_memory_test
conversation_memory_test: 24 checks passed
```

The full engine also builds and runs here (CUDA 13.2, sm_89, Qwen3.8-Flash-Next IQ3_S, 512K context, resident experts) on this branch.

## Not covered

`--session-save-reclaim` (#1573) frees retained K/V and unpinned checkpoints on a failed SAVE and retries; that path has the same "freed heap is not visible to `/proc/meminfo`" assumption and would benefit from the same `release()` between its steps. I left it alone here to keep this PR to the parking gate — happy to follow up if you want it.

Supersedes #1394.

En el sitio

Enlaces a install, modelos, releases.