Pull requests / #2917

#2917 Fix Qwen3.6 hybrid staging allocation on small GPUs

open · @CC-David-CC · 0 comments · View on GitHub

AMD / HIPNVIDIA / CUDA

Description

Release optional packed Q8 copies when hybrid staging and CPU-output scratch cannot fit within the VRAM reserve. This fixes Qwen3.6 Q4 startup on the tested Tesla P4. Explicit expert budgets, slots and KEEP_PACKED remain respected.

One source file only. Validated on CUDA 12.0 with the separate compiler fix in #2913: startup and generation passed; seven responses matched the manual workaround. HIP was not compiled.

Related on this site

Quick links to related pages.