Pull requests / #3121
#3121 conversation cache: opt-in overwrite growth for snapshot buffers
open · draft · @JumpEggKiller · 0 コメント · GitHub で見る
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
本文
## Summary Parking a long conversation grows host K/V buffers and then overwrites the new bytes with the snapshot copy. Add an opt-in overwrite growth path to avoid initializing those bytes twice. Only `STRATA_SNAPSHOT_NOFILL=1` enables it; the default still fills new bytes. Related: #2264. ## What changed - Store each `ConversationBuffer` segment in owned raw byte storage while preserving value semantics, address stability and ordinary resize filling. - Add `resize_for_overwrite` and use it at the snapshot capture call site under the opt-in switch. Keep the existing cache policy and snapshot format. - Add host tests for full overwrite, retained prefixes, segment boundaries, copy/move, allocation failure, failed transfer callbacks and discard/retry. - Include sanitized historical v0.1.45 park timings and their limits in `bench/results/conversation-overwrite/`. ## Validation and remaining work Standalone buffer tests on this main-based candidate: 92 checks pass with MSVC 19.44, C++17, optimized build. A separate v0.1.44.1 backport builds CUDA sm_86 and passes 92 buffer checks, 4,219 cache checks and 36 checkpoint split allocation-failure cases. The CSV is historical evidence from one patched v0.1.45 binary and two RTX 3090 cards at a 262,144-token context. It includes a slower overwrite run under host contention; it is not a current-main or backport benchmark. The README documents exactly what is timed and omitted. No model, private path, config, prompt, binary or credential is included. **Draft: not ready for review.** Full current-main CUDA, HIP and SYCL builds remain outstanding. HIP/SYCL toolchains are unavailable on this host. Fresh same-binary backport A/B, prefix edits, two alternating long histories, HTTP error recovery, and a NOFILL-enabled 3-hour/50-request soak are not yet complete. No production binary has been replaced. There is no claim that temperature zero makes outputs deterministic or that all failure modes have been covered.
関連リンク
インストール・モデル・リリースへの站内リンク。