反馈 / #1962

#1962 [Feature]: Share identical KV prefix blocks across concurrent conversations

open · @shoucandanghehe · 1 评论 · 去 GitHub 看

说明

## What do you want to do?

Allow concurrent conversations to reference the same physical KV blocks when they share an identical prefix, instead of storing a separate copy in every batch slot. The goal is vLLM-style prefix-block sharing, especially for multi-agent workloads with long common system prompts, tool definitions, or conversation history.

### Why this matters

Reusing prefix computation and sharing prefix storage are different optimizations. Strata can reuse an already-computed prefix, but separate concurrent slots still have their own KV storage. With KV streaming, this duplication consumes host RAM as well as the per-slot device state.

For example, if four agents branch from the same 50K-token prefix, the desired behavior is to store that prefix's KV once and let all four conversations reference it, while allocating their differing suffixes separately. This would reduce duplicate memory use and make more useful concurrency possible on memory-constrained machines.

### Requested behavior

- Share compatible, immutable prefix blocks across concurrent conversations.
- Keep each branch's new suffix and mutable execution state independent.
- Release a shared block only when no conversation or retained cache entry still references it; use copy-on-write or an exclusive tail block where a branch needs to diverge.
- Match the full prefix and relevant model/execution identity, including image inputs where applicable, rather than treating identical text fragments as interchangeable KV.

The primary request is physical KV deduplication in RAM, not a requirement to first implement a complete GPU PagedAttention stack. Sharing QSA KV should also preserve isolation of GDN/PLE, indexer, and checkpoint state.

### Related work, but a different goal

- #1614 provides a shared pinned-RAM KV allocation pool and on-demand chunks. Sharing an allocator does not by itself make identical prefixes share physical blocks.
- #1164 copies a prefix from a parked conversation while preserving the original history. This helps reuse without taking over that conversation, but still copies the KV.
- #1938 proposes limiting how much prefix each parked conversation retains, rather than deduplicating that prefix across conversations.
- [This discussion in #52](https://github.com/Niko1221/Strata/pull/52#issuecomment-5898751157) already describes page-level prefix sharing and reference counting in a contributor's own engine.

Would cross-conversation physical prefix sharing be in scope for Strata, possibly as a follow-up to the shared KV pool?

本站相关内容

相关页面的快捷入口。