贡献 / #1979

#1979 docs: Huihui abliterated UD-IQ4_XS, measured against Unsloth's and both OrcaRouter files

open · @fuad00 · 0 评论 · 去 GitHub 看

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

说明

Adds `docs/HUIHUI.md` (and a pointer in `docs/MODELS.md`, next to the OrcaRouter one): a manual workflow for
huihui-ai's abliterated Qwen3.8-Flash-Next in Unsloth's UD-IQ4_XS layout, in the style of `ORCA.md`. No code changes.

- Pinned revision and SHA-256s of the three shards. Shard 1 is byte for byte Unsloth's; shards 2/3 have Unsloth's
  sizes and the abliterated bytes. The pack has the same expert formats as Unsloth's UD-IQ4_XS.
- Packing is setup's own for Unsloth (`iq_pack.py --compat-bf16`); the config is a copy of setup's
  `strata-unsloth-ud-iq4_xs.json` with five values changed. `update.sh` keeps it up to date.
- The model card suggests patching `verify_sha256`; the doc says this path does not need it.

Measured on 2x RTX 4090 + Xeon Silver 4309Y + 188 GB RAM, engine 0.1.41, the same config for every model (layer split
24 + `--trim-stage-weights`, `parallel 4`, 262K context), fresh start + warm-up + 5 rounds, medians, nothing else on the
server:

| | Unsloth UD-IQ4_XS | Huihui UD-IQ4_XS | OrcaRouter IQ4_XS | OrcaRouter Q5_K_M |
| --- | ---: | ---: | ---: | ---: |
| Decode, tok/s | 112.8 | 110.5 | 107.8 | 87.4 |
| Prompt 30K, tok/s | 4,032 | 4,036 | 3,727 | 2,284 |
| 4 concurrent, total tok/s | 189.2 | 189.4 | 185.9 | 83.4 |
| base64 set (4 strings x enc/dec x 3) | 21/24 | 17/24 | 12/24 | 16/24 |

The task sets are small and their run-to-run spread is large; the doc says so and calls them smoke tests.

本站相关内容

相关页面的快捷入口。