Pull requests / #2443
#2443 conversation cache: evict redundant holders without losing pinned prefixes
open · @KeyanHu-git · 0 Kommentare · Auf GitHub
AMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Beschreibung
## Title conversation cache: evict redundant holders without losing pinned prefixes Fixes #2255. ## Summary When every parked conversation holds a pinned prefix, `evict_oldest()` refuses every entry. Several branches sharing one pin can fill the cache and prevent later branches from being parked, even with byte budget available. Allow eviction of redundant holders while preserving at least one holder of every pinned prefix. Thanks to @bennizone for reporting and diagnosing the problem; this also covers multiple distinct pins, where keeping any one pinned entry is insufficient. ## What changed - In `conversation_cache.hpp`, keep the existing preference for the oldest unpinned entry. If none exists, evict the oldest entry only when every pin it holds has another pinned holder with matching token IDs, image identity and steering mode. - Keep refusing eviction when each remaining entry is the sole holder of a pin. An older entry may need to survive because it holds another unique pin. - In `conversation_cache_test.cpp`, cover repeated holders, independent pins, multi-pin entries, image/steering identity, slot and byte pressure, and eviction accounting. No new options or budget changes. The existing caller path that skips parking siblings resuming exactly at the pinned prefix is unchanged. ## Extra Notes ### Regression tests Base: `296b59d2`. All **4,242 checks pass** with macOS Clang (ASan/UBSan), Windows AMD Clang, Linux GCC 14 and Intel icpx. ```sh c++ -std=c++20 -Wall -Wextra -Werror -Iinclude src/core/conversation_cache_test.cpp -o conversation_cache_test ./conversation_cache_test ``` The same regression tests fail on unmodified upstream when parking a fourth holder of one pin. An any-pin fallback fails the independent-pin case by evicting pin A's sole holder while pin B remains. ### Real-model comparison Windows, Ryzen 5 7600X / RX 7900 XTX, Coder IQ1_M with MTP (`--spec 4`). Both HIP binaries were built from the same base, with only this patch differing. Each arm ran 15 fixed requests: two pin-only controls, one common-history request, six branches, and six revisits. Candidate inputs use the same token IDs and baseline-generated history. | Observation | Upstream | Patched | | --- | ---: | ---: | | Parking refusals | 8 | 0 | | Branch snapshot restores | 0 | 6 | | Tokens actually prefilled across six revisits | 1,778 | 90 | All 15 paired outputs are token-identical. All six restored MTP draft-KV checks pass with `STRATA_SNAPSHOT_VERIFY=1`. Matched settings: conversation cache 4,096 MiB / 3 slots, prompt cache 6 / interval 16,384, resident budget 8 GiB, adaptive swaps and PCIe fraction both 0, `STRATA_IQ_MT_MIN=1`, at most 16 output tokens per request. This is a native-engine GEN test with synthetic multi-turn inputs. A preceding single-turn control did not trigger the bug. Diagnostic timings are not used to claim latency or throughput improvements. ### Backend builds | Backend | Validation | | --- | --- | | HIP | Full gfx1100 engine build and the model comparison above | | CUDA | Full Linux build, nvcc 13.0.88, sm_120; version startup succeeds | | SYCL | Full Linux build, oneAPI 2026.1.1, SPIR-V; no available SYCL platform on the build host, so version startup fails during device initialization | CUDA and SYCL model inference was not tested. ### Cost The additional scan runs only after the unpinned-entry search fails. Its cost depends on entry count, checkpoint count and prefix lengths. Policy-only probes with 32 distinct 100K-token prefixes took milliseconds per scan; production latency impact has not been measured. The patch keeps complete snapshots and does not introduce a separate prefix-storage mechanism.
Mehr auf der Site
Links zu Install, Modellen, Releases.