Issues / #1957

#1957 [Question] Parameters to run Q4-K-M-XL on 48GB VRAM + 64GB RAM?

open · @MaximS · 0 comentários · No GitHub

NVIDIA / CUDA

Descrição

Please, what parameters should I use to run Q4-K-M-XL on 48GB VRAM (CUDA) + 64GB RAM?
Will entire model split between VRAM and RAM and use expert swap between VRAM and RAM instead of reading it from SSD? Do I correctly understand the Low-RAM mode?
Or I need to wait for #1533 draft for it?

No site

Links install, modelos, releases.