Issues / #2508

#2508 KV compaction and hot-swappable KV sessions (two proposals)

open · @shahrokhzargarpour · 1 commentaires · Sur GitHub

Models & quantsWindows

Description

Two forward-looking KV ideas, split clearly from what we already run.

## What already exists (our fork)

The conversation-cache disk tier we run saves by stage as content-addressed segments, dedups the `Sys` between conversations (same `head = H(model||config)`), and has a prefix restore ladder. Measured on Windows: reload of a 171.5k context in 657 ms vs 158.6 s cold, and 1,294 real restores over 8 days with about 90.5 M tokens not re-prefilled. A PR for this disk tier is being prepared from our fork.

## (a) KV compaction

Today the tier saves per stage, but the stationary KV components - the system prompt and fixed headers - are still carried inside every image. The missing piece: compact those stationary components as independent, deduplicated KV segments (their own content-addressed blocks) instead of dragging them through every park/restore. Related: #1090 and #1331 (system-prompt prefix on disk), #1962 (share identical KV prefix blocks across concurrent conversations).

## (b) Continuous KV sessions

Store KV as detachable fragments that can be dropped and re-attached in place - a dynamic context window guided by RAG (fragments retrieved by embedding + reranking) instead of re-prefilling. Related: #2351 (prefix mount, splices, checkpoint ladder), #2431 (prefix reuse falls back to root-ck), #1269/#1511/#1164 (session restore and prefixes), and #2434 (KV page-pool policy measurement).

## Not proven

- The stationary-KV compaction is not implemented or measured.
- No RAG-guided fragment retention/retrieval has been built or benchmarked - section (b) is aspirational, not a claim.
- These are Windows-only measurements; the maintainer noted there is no Windows test box on their side (#2185/#1412).

Sur le site

Liens install, modèles, releases.