Issues / #2415

#2415 [Feature]: kv_stream page-pool eviction: measure CLOCK against selection-aware LRFU before changing anything

open · @MrKop-git · 0 commentaires · Sur GitHub

BenchmarksSetup & installAMD / HIPModels & quants

Description

## What do you want to do?

With `--kv-resident`, a streamed QSA layer keeps its full K/V in pinned RAM and a page pool in VRAM. `kv_stream_resolve` makes every block the selection names resident, and when the pool is full a CLOCK (second-chance) sweep picks the victim. The QSA indexer already names the blocks each step reads — the hit signal is right there, and `kv_stream_counters` already reports misses / lookups / calls.

I would like a measured eviction-policy sweep for that pool: replay real selection traces through CLOCK and through a selection-aware LRFU keep score (each hit adds the selection's weight, every step decays: `KS = λ·KS + hit`, λ around 0.5), and adopt the winner as the default only where it measurably wins. The same "eviction policy is a measured question" approach as #1089 takes for the expert cache's `admit`.

## Why

Fewer RAM→VRAM block re-reads is less PCIe traffic and shorter stalls on long contexts where KV streaming is on (64K and up, as setup chooses). The page pool is only a cache — `kv_stream_parity` proves the streamed readers see exactly the resident values — so a policy change cannot change any answer; it is purely a throughput question.

## How I would check it

- an offline bench along the lines of `tools/replay_cache_policy.py`, replaying dumped selection traces through both policies and reporting misses and bytes re-read;
- a live A/B on a named card and model with the same prompts, reporting miss rate and wall time per request, and what was not tested;
- an opt-in switch first (`--kv-evict clock|lrfu`, default `clock` unchanged); a default change only with the numbers in the PR.

## Context

Outside evidence, for motivation only: LRFU keep scores with a decay factor are what ActKV (arXiv:2609.31395) uses for agentic KV retention, and kvmem 0.18.0 ships LRFU (lambda=0.5, top-frac=0.9) as its default history weighting. Whether that beats CLOCK on this engine's QSA selections is exactly the measurement this issue asks for.

Sur le site

Liens install, modèles, releases.