Pull requests / #2917

#2917 Fix Qwen3.6 hybrid staging allocation on small GPUs

open · @CC-David-CC · 0 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDA

Beschreibung

Release optional packed Q8 copies when hybrid staging and CPU-output scratch cannot fit within the VRAM reserve. This fixes Qwen3.6 Q4 startup on the tested Tesla P4. Explicit expert budgets, slots and KEEP_PACKED remain respected.

One source file only. Validated on CUDA 12.0 with the separate compiler fix in #2913: startup and generation passed; seven responses matched the manual workaround. HIP was not compiled.

Mehr auf der Site

Links zu Install, Modellen, Releases.