Pull requests / #2526
#2526 Preserve decoded batch-slot conversations before slot reuse
open · draft · @hotschmoe · 0 comments · View on GitHub
Description
After a 256-token batched reply, admitting another conversation could overwrite its slot while the RAM cache held only the original prompt. Returning to that chat then replayed the reply, or the entire prefix if stale cache entries had displaced it. Park an idle text slot before either overwrite path, using its own sessions, checkpoints and per-slot draft K/V through the existing cache budget. Carry the slot checkpoint chain back into the main session so newer versions supersede older parked copies. Reuse one snapshot image buffer per stage across layers; save/restore still overwrites the same bytes. Image slots and MTP without a per-slot batch drafter retain their existing skip behavior. Latest text continuations are covered; arbitrary shorter branch endpoints remain outside this guarantee. This fix makes no latency guarantee and is independent of #2511. Validation: - Independent main-based branch: 4,271 CPU cache checks and SYCL generate syntax check passed in a CPU-only container. - Same patch on the EP/PUSH/accounting test stack: four 199,501-token chats, two active 256-token replies, then four forced RAM returns. Each return reused 199,839 held tokens, read 31 new tokens and replayed zero retained reply tokens. Identity, concurrent arithmetic, reasoning, tool calls, normal teardown, both-card health and ten compiled collective iterations passed. [Measured configuration and scope](https://github.com/hotschmoe/b70_ai_things/blob/b9fd3cb/docs/20261014_flashnext_strata_optimization.md#s20-four-long-conversations-recover-from-ram-2026-10-15). - CUDA mirror: source reviewed; not compiled or run. The full GPU evidence above is from SYCL with the separate accounting overlay, rather than an independent-branch GPU rebuild.
Related on this site
Quick links to related pages.