Pull requests / #3181

#3181 [Other] setup: pass the family's pack args to the low-RAM experts.bin run (Gyro needs --compat-bf16)

open · @victoralcazardev · 0 comments · View on GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

Problem and result: on a PC where setup picks the low-RAM mode, a Gyro install (`--family gyro`) stops at the second pack run. Setup packs twice: first `iq_pack.py ... --compat-bf16` (with the family's `pack_args`), then `iq_pack.py ... --experts-bin` for low-RAM mode. The second call does not pass `pack_args`, so `iq_pack.py` rejects Gyro's non-BF16 hc tensors and exits 1:

```
output_hc_down.weight is IQ4_NL, but the engine requires BF16; use --compat-bf16
blk.0.hc_attn_down.weight is Q6_K, but the engine requires BF16; use --compat-bf16
... and 356 more
  [X]  command failed (exit 1): python
```

The first run has already written `native_experts.txt`, so a rerun skips it and fails at the same point every time; the install cannot finish. With this change both runs pass the family's `pack_args`.

Approach: one-line change in `setup.py`: the `--experts-bin` run also gets `*fam.get("pack_args", [])`, like the pack run just above it. Families without `pack_args` get the same command as before.

Validation:
  Passed: reproduced on hardware: RX 7900 XTX (gfx1100), Linux, 31 GiB RAM, clone at v0.2.0-beta-4-g4caf857c, `./setup.sh --setup --family gyro --model Gyro-S --backend hip --vision no --yes --no-start --context 262144 --kv int8 --vram-reserve-mib 3072`: stopped with the error above. Ran the same command setup uses, with `--compat-bf16` added (`tools/iq_pack.py --gguf <Gyro-S gguf> --out <pack> --experts-bin --compat-bf16`): wrote `experts.bin: 48 layers, 24.02 GiB`. A rerun of setup then finished ("All set", start script written).
  Passed: `python -m unittest tools.test_setup_gyro` (13 tests OK). The new `GyroLowRamPack` test fails on main (`'--compat-bf16' not found in [... '--experts-bin']`) and passes with this change.
  Passed: `python -m unittest discover -s tools -p 'test_setup_*.py'`: 687 tests. The same 50 failures occur on main (3fa1b524) without this change (48 in `test_setup_golden`, 2 in `test_setup_engine_hash`); the list of failures is identical with and without it.
  Not run: setup.py with this patch end to end on hardware (only the patched command, run by hand, as above); Gyro-S inference on gfx1100; no engine or backend code changed, so no GPU builds.
Limits: none known. The Unsloth family also has `pack_args`, but it does not reach this branch.

Related on this site

Quick links to related pages.