Issues / #1957

#1957 [Question] Parameters to run Q4-K-M-XL on 48GB VRAM + 64GB RAM?

open · @MaximS · 0 Kommentare · Auf GitHub

NVIDIA / CUDA

Beschreibung

Please, what parameters should I use to run Q4-K-M-XL on 48GB VRAM (CUDA) + 64GB RAM?
Will entire model split between VRAM and RAM and use expert swap between VRAM and RAM instead of reading it from SSD? Do I correctly understand the Low-RAM mode?
Or I need to wait for #1533 draft for it?

Mehr auf der Site

Links zu Install, Modellen, Releases.