Pull requests / #2265

#2265 Conversation cache: --conversation-cache-park-gain N (opt-in, default 0): a turn that grew a few hundred tokens no longer rewrites a whole 3 GB snapshot

open · @Krypto-Whitehat · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDASecurityWindowsLinux

Descripción

Issue: Resolves #2264

## Summary

`--conversation-cache-park-gain N` makes parking rate-limited by growth, opt-in and **default 0 = park every turn, byte-identical to today**. A conversation that advanced fewer than N tokens since the copy it would supersede keeps that copy: it still holds this conversation's prefix (a conversation only appends), and re-parking writes the whole snapshot only for the next turn to drop it again.

The report behind this measured 1,418 successful parks for one long single conversation, 2,251.9 GB of host copy written and 105.8 GB (4.7 %) ever read back, with 425 superseded copies dropped and 1,548 s of copy time on the request path. With the gate on it measured 3 park events and 3.26 GB (-66.7 %), the remaining copy being the conversation's first park.

The default stays off because the measurement is from one rig. Deciding whether it should be on by default is the maintainer's call, and nothing here forecloses it.

## What changed

- `include/strata/core/conversation_cache.hpp`: one new const accessor, `parked_prefix_tokens(ids, images, checkpoints, cvec)`. It reports, in tokens, the longest parked entry the outgoing state supersedes - exactly the entries `drop_superseded()` would remove, found the same way `drop_superseded()` finds them (deepest checkpoint the outgoing chain still holds), so the gate can never name a different entry than the one a park would drop.
- `src/program/generate.cpp`, the park decision (line 10445 at `a0116b27`): the existing condition gains one term, `park_worth`. With the default 0 that term is constantly true and the line behaves as before.
- `src/program/generate.cpp`: the option, the `--conversation-cache-park-gain N` parse (alongside the other `--conversation-cache-*` flags, same `INT32_MAX` bound as `--conversation-cache-slots`), and the help text.
- `src/core/conversation_cache_test.cpp`: seven checks for the accessor - an empty cache, parking a conversation, the entry the chain supersedes, another conversation sharing only the root, the other steering mode, an appended tail, and an entry without checkpoints.

Left alone on purpose: the default (`0`, so no behaviour change without the flag), the copy path itself, `drop_superseded()` and the K/V retention, the physical-RAM admission gate, and every log line. This is one flag, one accessor and one term in one condition.

## Extra Notes

- Checked by build: `cmake -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89` with nvcc 13.4.92 and MSVC 14.44. The `strata` target compiles and links with 0 errors (Windows CUDA, sm_89, so the Linux-only paths of this file are not what that compiles). `conversation_cache_test` builds and passes: 4,226 checks, exit 0.
- The unit test earned its keep: it failed the first assertion because I had expected the entry's live length (6) where the accessor reports the deepest checkpoint (5). `drop_superseded()` keys on the checkpoint, so 5 is the right answer and the expectation was wrong.
- Not tested here: HIP and SYCL (no toolchain on this machine), and **no decode run on this box** - there is no model and no matching rig. Every tok/s number in the issue comes from the reporter's own A/B on an RTX 5090 Laptop, not from a run of this patch. What is verified here is that the change compiles and that the accessor names exactly what a park would supersede.
- The gate is a policy decision with a real trade-off the maintainer should weigh: a conversation that rewrites (edits an earlier turn) is not a prefix of its parked copy, so with the gate on, such a conversation keeps an entry it cannot reuse until it grows past the threshold. That is the same loss the cache already accepts for `drop_superseded`, and it is why this ships opt-in.

En el sitio

Enlaces a install, modelos, releases.