Issues / #429
#429 setup: UD-Q4_K_XL is offered (and not blocked) on --backend hip, though docs/UNSLOTH_Q4.md says it needs an NVIDIA GPU
closed · @jkuepker · 1 comentarios · En GitHub
Setup & installServer & APIAMD / HIPModels & quantsDocumentationLinux
Descripción
On an AMD-only PC, `./setup.sh --check --backend hip` lists the experimental Unsloth model as fitting: ``` UD-Q4_K_XL needs ~48 GB RAM: EXPERIMENTAL, fits with 38 GiB of its experts in RAM, the rest read from the SSD ``` `docs/UNSLOTH_Q4.md` says it needs "48 GB of RAM or more and engine 0.1.32 or newer ... and an NVIDIA GPU." Reading `setup.py` at v0.1.32: the UD-Q4_K_XL checks before the download cover the RAM size (`MODELS[model]["ram_gb"]`) and the engine version (`UNSLOTH_ENGINE`), but nothing checks the backend. So an AMD user can pick it and start the 111 GB download. I did not try the download, so I don't know whether the HIP engine actually runs it (`--compat-bf16`, resident-budget mode). **Expected:** one of these: - If HIP isn't supported: `--check` marks UD-Q4_K_XL as "NVIDIA only" when the backend is hip, and setup refuses it before downloading. This matches how the RAM and engine-version checks run first. - If HIP is meant to work: update `docs/UNSLOTH_Q4.md` and say it is untested on AMD. **Environment** ``` Strata v0.1.32 (tag c499bd1), Linux, Ubuntu 24.04.4, kernel 7.0.0-34 GPU: Radeon AI PRO R9700 32 GB (gfx1201), single card CPU: i5-12600K (AVX2, no AVX-512), 62 GB DDR5 Setup: ./setup.sh --backend hip --family coder --model IQ1_M (no system ROCm) ROCm: TheRock 7.10.0a20251120 wheels (gfx120X-all) installed by setup into .venv; hipBLASLt 1.2.0 Server: serve/server.py with setup's generated config (max-context 131072, kv int8, spec 4 + MTP) ```
En el sitio
Enlaces a install, modelos, releases.