#1938 Proposal: Add a Per-Slot Cached-Prefix Limit to Conversation Cache

Description

In memory-constrained setups (4–8 GiB for conversation cache), full-slot KV snapshots(262144 context) are too large: INT8 KV ≈ 3.3 GiB/slot, FP16 KV ≈ 6 GiB/slot. As a result, only a few slots fit, and new ones evict existing ones.

Add an option like `--conversation-cache-slot-max-preserve N` (or shorter: `--conversation-cache-slot-prefix N`) to cache only the first N tokens of each parked slot. Later updates are not cached. This lets more slots retain a useful common prefix and improves cache utilization.

I recently ran into the issue with hermes agent + strata: hermes agent has a background review that reuses the main conversation branch and triggers a fork (according to AI analysis), causing subsequent main-conversation turns to fail to find a prefix. The experience is painful—every background review forces a long re-prefill wait. 

Of course, if the issue isn't the slot being evicted into the conversation cache, but rather the fork occupying the main conversation's slot and advancing past several checkpoints, thereby causing the main conversation to lose its prefix, that would be a separate issue.