Pull requests / #1979
#1979 docs: Huihui abliterated UD-IQ4_XS, measured against Unsloth's and both OrcaRouter files
open · @fuad00 · 0 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsDocumentation
Beschreibung
Adds `docs/HUIHUI.md` (and a pointer in `docs/MODELS.md`, next to the OrcaRouter one): a manual workflow for huihui-ai's abliterated Qwen3.8-Flash-Next in Unsloth's UD-IQ4_XS layout, in the style of `ORCA.md`. No code changes. - Pinned revision and SHA-256s of the three shards. Shard 1 is byte for byte Unsloth's; shards 2/3 have Unsloth's sizes and the abliterated bytes. The pack has the same expert formats as Unsloth's UD-IQ4_XS. - Packing is setup's own for Unsloth (`iq_pack.py --compat-bf16`); the config is a copy of setup's `strata-unsloth-ud-iq4_xs.json` with five values changed. `update.sh` keeps it up to date. - The model card suggests patching `verify_sha256`; the doc says this path does not need it. Measured on 2x RTX 4090 + Xeon Silver 4309Y + 188 GB RAM, engine 0.1.41, the same config for every model (layer split 24 + `--trim-stage-weights`, `parallel 4`, 262K context), fresh start + warm-up + 5 rounds, medians, nothing else on the server: | | Unsloth UD-IQ4_XS | Huihui UD-IQ4_XS | OrcaRouter IQ4_XS | OrcaRouter Q5_K_M | | --- | ---: | ---: | ---: | ---: | | Decode, tok/s | 112.8 | 110.5 | 107.8 | 87.4 | | Prompt 30K, tok/s | 4,032 | 4,036 | 3,727 | 2,284 | | 4 concurrent, total tok/s | 189.2 | 189.4 | 185.9 | 83.4 | | base64 set (4 strings x enc/dec x 3) | 21/24 | 17/24 | 12/24 | 16/24 | The task sets are small and their run-to-run spread is large; the doc says so and calls them smoke tests.
Mehr auf der Site
Links zu Install, Modellen, Releases.