Pull requests / #2900

#2900 Community Strix Halo bench: UD-Q4_K_XL vs UD-IQ4_XS on 121 GiB UMA (gfx1151)

open · @ianmin2 · 0 comentários · No GitHub

BenchmarksSetup & installAMD / HIPModels & quantsDocumentation

Descrição

### What

Community Strix Halo (gfx1151) results comparing the two Unsloth quants on 121 GiB of unified memory: **UD-Q4_K_XL versus UD-IQ4_XS**, speed *and* correctness, on a machine running as a desktop with a live interactive workload rather than an isolated benchmark host.

`docs/STRIX_HALO.md` §"Measured" already gives the fast-settings A/B for these same two packs; this is the user-facing **"which one should I actually run"** question, where the answer is not the same on both axes.

### Machine

EVO-X2, Ryzen AI Max+ 395 / Radeon 8060S (**gfx1151**), 128 GB installed UMA (`MemTotal` 121.2 GiB), Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic`, in-tree `amdgpu` (no DKMS), power mode `performance`. ROCm **7.14.1** (`/opt/rocm/core-7.14`, HSA/ROCr 1.21.0, `libamdhip64.so.7.14.60850`), bundled gfx1151/100401 tuning table, **no swap**, no `HSA_OVERRIDE_GFX_VERSION`, no BIOS or kernel changes.

### Results

| | UD-IQ4_XS (v0.1.43, budget 55) | UD-Q4_K_XL (v0.1.44.1, budget 71) |
|---|---|---|
| ctxbench standard / hard | 95.6 % / 98.4 % | **96.7 % / 99.5 %** |
| decode: code / prose / 128 K | **69.3 / 50.8 / 38.1** t/s | 67.0 / 41.3 / 35.8 t/s |
| prefill 4 K / 16 K / 128 K | 932 / 1,187 / **1,212** | **1,067 / 1,225** / 1,172 |
| `MemAvailable` during the run | 37–38 GiB | 14 GiB |
| GGUF on disk | 93,682,584,224 B (87.25 GiB) | 111,334,654,784 B (103.69 GiB) |

**The larger quant is more correct and slower at decode.** On 121 GiB you keep either the headroom or the accuracy, not both: Q4_K_XL leaves about 14 GiB for the OS and everything else, and with no swap an over-claim OOMs rather than degrades. `--resident-budget-gib` stays explicit for the reason already documented in `STRIX_HALO.md` (#1715).

### Please read the correctness number with its failure table

I checked the claim behind "96.7 % vs 95.6 %" and it does not mean broad reasoning superiority. Both arms fail the **same two tasks**; B fails them less often:

| Battery | Arm | failed rows | concentrated in |
|---|---|---|---|
| standard | UD-IQ4_XS | 8 | `reasoning/optimisation` ×7, `game_trace/three_files` ×1 |
| standard | UD-Q4_K_XL | 6 | `reasoning/optimisation` ×6 |
| hard | UD-IQ4_XS | 3 | `science_ctx/kinetic_energy` ×2, `synthesis/count_limits` ×1 |
| hard | UD-Q4_K_XL | 1 | `science_ctx/kinetic_energy` ×1 |

`reasoning/optimisation` accounts for 7 of A's 8 and all 6 of B's standard failures, every one with the identical signature `got='None' want='102'` + `RULES BROKEN: answer_line` — output-format compliance on a flaky task, repeated across context levels, carrying most of the 1.1-point gap. The one clean signal is at **level 0** (no reference context at all), where A failed that task and B answered it in 50.3 s, which cannot be a retrieval effect — but it is one task on a task both arms flake on, so the report calls it a hint worth a re-run rather than proof.

### Caveats, stated in the report rather than smoothed over

- The arms differ in **engine version** (0.1.43 vs 0.1.44.1, five weeks) and in spec settings, so speed deltas are quant-**plus**-config.
- The smaller-quant arm was served directly with nothing else resident; the larger-quant arm ran behind a proxy **with live interactive traffic** that could not be detached. `pp`/`tg` are engine-reported so the proxy should not distort them, but the uncontrolled load is real.
- **Single run per long row** (short rows 3×). Not an isolated OS.
- `pp245k` fails in **both** arms at ~267.3 K prompt tokens against a 262,144 window — a context ceiling refused before inference, explicitly **not** an OOM. Nothing here characterises above 128 K.
- **Not measured:** needle recall, parallel/QFUSE, thermal telemetry, build-manifest provenance. `TRIMMED.md` names these so the scope is not mistaken for the fuller gfx1151 reports in this directory.
- **Reproducibility limit:** the load driver and the correctness harness are submitter-local scripts that are *not* published with this report (and are not part of Strata), so these exact numbers cannot be re-derived from the repository alone. Happy to supply them on request. The raw outputs are **SHA-256 pinned in `REPRODUCE.md`** — a checksum against private artefacts, not committed evidence — and are not in the commit.

### Files

4 files, docs and data only — no engine changes. New `bench/results/2026-10-16-community-gfx1151-ud-q4kxl-vs-iq4xs-0.1.44/{README.md,REPRODUCE.md,TRIMMED.md}` plus one index row in `bench/results/COMMUNITY.md`.

No site

Links install, modelos, releases.