Issues / #429

#429 setup: UD-Q4_K_XL is offered (and not blocked) on --backend hip, though docs/UNSLOTH_Q4.md says it needs an NVIDIA GPU

closed · @jkuepker · 1 Kommentare · Auf GitHub

Setup & installServer & APIAMD / HIPModels & quantsDocumentationLinux

Beschreibung

On an AMD-only PC, `./setup.sh --check --backend hip` lists the experimental Unsloth model as fitting:

```
UD-Q4_K_XL needs ~48 GB RAM: EXPERIMENTAL, fits with 38 GiB of its experts in RAM, the rest read from the SSD
```

`docs/UNSLOTH_Q4.md` says it needs "48 GB of RAM or more and engine 0.1.32 or newer ... and an NVIDIA GPU."

Reading `setup.py` at v0.1.32: the UD-Q4_K_XL checks before the download cover the RAM size (`MODELS[model]["ram_gb"]`) and the engine version (`UNSLOTH_ENGINE`), but nothing checks the backend. So an AMD user can pick it and start the 111 GB download. I did not try the download, so I don't know whether the HIP engine actually runs it (`--compat-bf16`, resident-budget mode).

**Expected:** one of these:
- If HIP isn't supported: `--check` marks UD-Q4_K_XL as "NVIDIA only" when the backend is hip, and setup refuses it before downloading. This matches how the RAM and engine-version checks run first.
- If HIP is meant to work: update `docs/UNSLOTH_Q4.md` and say it is untested on AMD.

**Environment**

```
Strata v0.1.32 (tag c499bd1), Linux, Ubuntu 24.04.4, kernel 7.0.0-34
GPU: Radeon AI PRO R9700 32 GB (gfx1201), single card
CPU: i5-12600K (AVX2, no AVX-512), 62 GB DDR5
Setup: ./setup.sh --backend hip --family coder --model IQ1_M (no system ROCm)
ROCm: TheRock 7.10.0a20251120 wheels (gfx120X-all) installed by setup into .venv; hipBLASLt 1.2.0
Server: serve/server.py with setup's generated config (max-context 131072, kv int8, spec 4 + MTP)
```

Mehr auf der Site

Links zu Install, Modellen, Releases.