Issues / #1957

#1957 [Question] Parameters to run Q4-K-M-XL on 48GB VRAM + 64GB RAM?

open · @MaximS · 0 comments · View on GitHub

Description

Please, what parameters should I use to run Q4-K-M-XL on 48GB VRAM (CUDA) + 64GB RAM?
Will entire model split between VRAM and RAM and use expert swap between VRAM and RAM instead of reading it from SSD? Do I correctly understand the Low-RAM mode?
Or I need to wait for #1533 draft for it?

Related on strata.com

Install Strata
Model sizes & RAM
Release notes

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.