Issues / #1572
#1572 [Feature]: Optionally reclaim disposable host caches when session SAVE needs RAM
open · @tuandat3019 · 0 comentarios · En GitHub
Descripción
### What do you want to do? Let an explicitly enabled session SAVE reclaim reconstructible host caches before refusing its RAM preflight. This is useful on a 32 GiB PC when saving a long conversation to unload the LLM and use the GPU for another local workload. The current admission check correctly protects the physical-RAM floor, but parked conversations, retained K/V and non-selected checkpoints can occupy RAM that the file will not contain. Our local workaround releases those caches before SAVE; I would like to contribute a narrower, opt-in implementation that only evicts under measured pressure, preserves explicit pins and the checkpoint selected for the file, and checks the original floor again after each release. No lower RAM floor, format change, RESTORE change, process working-set trimming, or guarantee of saving under arbitrary pressure is proposed. Eviction is a tradeoff: later prefix reuse may require prefill, and released auxiliary caches cannot be rolled back if subsequent I/O fails. The live session and any previously published file must remain valid. I searched the existing session/disk-cache PRs, including #668, #1269, #1271, #1480, #1489 and #1529. This proposal addresses SAVE admission independently of disk spilling, streamed RESTORE or non-MTP support. Prepared against main `fb58e0d` (0.1.41), tested on Windows with RX 6800 16 GiB and 32 GiB system RAM. A draft PR will include CPU failure-injection/file-parity tests and HIP build/runtime results; CUDA build validation is not available on this machine.
En el sitio
Enlaces a install, modelos, releases.