Pull requests / #2631
#2631 Metal: the conversation cache keeps a conversation across a side request (cause 2 of #2502)
open · @PiMatyja · 0 comments · View on GitHub
AMD / HIPNVIDIA / CUDAModels & quants
Description
## Summary Fixes cause 2 of #2502 (cause 1 is #2503, which this needs). On a Mac, `--conversation-cache-mib` never keeps a conversation, so a short side request (a title call) makes the next turn re-read the whole prompt. This makes the cache save and restore the Metal decode window's own K/V and state. No new flag; without `--conversation-cache-mib` nothing changes. CUDA, HIP and SYCL compile none of it. ## Extra Notes M2 Max, 64 GB, IQ3_XXS resident, with #2338 and #2503, #2502's steps (17K-token prompt, two follow-ups, a title request, turn 4), 3 rounds with and without MTP: | turn 4 | reused | time to first token | | --- | ---: | ---: | | before | 0 | 57-59 s | | this PR | 17,616 | 0.7 s | Same answer as a cold read in every run. Session files (`SAVE` / `LOAD`) are refused on Metal with a clear message. Builds on CUDA with no new warnings.
Related on this site
Quick links to related pages.