Pull requests / #2631

#2631 Metal: the conversation cache keeps a conversation across a side request (cause 2 of #2502)

open · @PiMatyja · 0 comments · View on GitHub

AMD / HIPNVIDIA / CUDAModels & quants

Description

## Summary
Fixes cause 2 of #2502 (cause 1 is #2503, which this needs).

On a Mac, `--conversation-cache-mib` never keeps a conversation, so a short side request (a title call) makes the next turn re-read the whole prompt. This makes the cache save and restore the Metal decode window's own K/V and state. No new flag; without `--conversation-cache-mib` nothing changes. CUDA, HIP and SYCL compile none of it.

## Extra Notes
M2 Max, 64 GB, IQ3_XXS resident, with #2338 and #2503, #2502's steps (17K-token prompt, two follow-ups, a title request, turn 4), 3 rounds with and without MTP:

| turn 4 | reused | time to first token |
| --- | ---: | ---: |
| before | 0 | 57-59 s |
| this PR | 17,616 | 0.7 s |

Same answer as a cold read in every run. Session files (`SAVE` / `LOAD`) are refused on Metal with a clear message. Builds on CUDA with no new warnings.

Related on this site

Quick links to related pages.