#2033 [Enhancement] Mark conversation cache MADV_COLD to use swap preferentially on Linux

Descrição

On a 64GB system running IQ3_S, there is no RAM left for a conversation cache. However, in long-running agentic workloads, this can lead to quite a bit of time reprocessing context (~30s on avg for me).

My suggestion is to mark conversation cache memory allocation as MADV_COLD, which on Linux indicates to the kernel that these allocations can be opportunistically moved to swap. The time taken to move cached convos from swap back to VRAM should be much faster. And this would work transparently vs disk-backed cache which has code complexity, staleness issues and requires manual operation via the slot save & load routes. The downside of course is that swap-backed cache isn't disk-backed.

I am not sure if even more can be done besides MADV_COLD, for example to outright allocate on swap or protect the limited RAM left being used between whatever strata puts into disk-backed page cache and other system processes.

I will try making the modification to see if the added memory pressure causes any issues.