反馈 / #1957

#1957 [Question] Parameters to run Q4-K-M-XL on 48GB VRAM + 64GB RAM?

open · @MaximS · 0 评论 · 去 GitHub 看

NVIDIA / CUDA

说明

Please, what parameters should I use to run Q4-K-M-XL on 48GB VRAM (CUDA) + 64GB RAM?
Will entire model split between VRAM and RAM and use expert swap between VRAM and RAM instead of reading it from SSD? Do I correctly understand the Low-RAM mode?
Or I need to wait for #1533 draft for it?

本站相关内容

相关页面的快捷入口。