Guides

Out of VRAM, what next

Drop one quant level first — Q5_K_M to Q4_K_M typically frees a third of the memory for a small quality loss. Shrinking context also helps, because the KV cache grows with it. Only then consider a smaller model or a second card.