Pull requests / #2925

#2925 Run Qwen3.6 A3B Q4 on 6 GB VRAM and 16 GB RAM

open · @CC-David-CC · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsLinux

Beschreibung

Adds opt-in `--low-memory` with a bounded GGUF-backed expert cache and unused MTP weights unloaded. Qwen3.6-35B-A3B UD-Q4_K_XL generated 128 tokens at ~4.5 tokens/s on a GTX 1060 6 GB with 16 GB system RAM.

Used `--low-memory --expert-ram-gib 6 --mtp 0 --max-context 2048 --prefill 256`. Linux/CUDA validated; HIP untested. CUDA 12.0 needs the separate compiler fix in #2913.

Mehr auf der Site

Links zu Install, Modellen, Releases.