#6450 [Engine] reduce host RAM peaks when alternating parked conversations

open · @G0-0000 · 0 comentarios · En GitHub

BenchmarksNVIDIA / CUDA

Descripción

## Problem

On a compatible whole-session RAM-cache take, switching A -> B can transiently hold the active host K/V allocation, B's incoming parked K/V, and a newly allocated or retained parked image for A. This makes two long alternating conversations require a third full host-KV allocation at the switch point even when the steady state fits.

This allocation pattern is **unconfirmed static analysis of the original path**, not a reproduced OOM report.

## Proposed scope

For a compatible ordinary whole-session RAM take, keep the active host K/V addresses fixed and exchange the outgoing/incoming K/V and scale bytes using one bounded scratch block. Reuse B's incoming parked allocation as A's outgoing parked allocation instead of constructing or retaining another full host-KV image.

Add `--conversation-cache-swap-block-mib`: 0 keeps the current path and is the upstream default; values 1-256 MiB enable the bounded exchange. Borrow/checkpoint-only, device-only, batch/split, explicit file restore and other layouts that cannot be validated keep their existing paths. No disk cache, compression, expert streaming or GPU-memory behavior is in scope.

The tradeoff is one additional copy per exchanged chunk in return for bounded scratch memory. Correct fallback is allowed only before the first exchanged byte; a post-write failure must not continue with mixed state.

## Local evidence

A focused implementation has passed conversation-cache/transfer/buffer tests and a CUDA 12.9 sm_70 build. On the local V100 service, four warmed long-conversation returns completed with their own prefix reuse and dedicated chunked-swap log evidence. These are functional checks, not a cross-platform or peak-RSS benchmark.

## Related discussions

- #189: retained K/V buffers and incremental parking.
- #2264 and #5895: related host-RAM pressure/copy reports; this proposal does not claim to solve every limit in those threads.
- #3351 / #1011: a different shared-pool/block-table approach.
- #4226 and #2510: disk streaming; their bounded file chunks are not this RAM-to-RAM exchange.

A draft PR will link this issue for scope discussion and list its exact validation and unrun backend checks.

En el sitio

Enlaces a install, modelos, releases.