#6421 [Benchmark] Ryzen 7 5700X, 64GB DDR4-2400, 2x RX 7800 XT 16GB (Ubuntu/ROCm) -- Qwen3.8-Flash-Next UD-IQ4_XS (256K, pinned 41.4 GiB, --spec 4)

open · @andrejlist · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Beschreibung

Follow-up benchmark to #6202 (`IQ3_S` vs `UD-Q4_K_XL`) on the exact same machine, testing **`unsloth/Qwen3.8-Flash-Next-GGUF` `UD-IQ4_XS`** (`87.2 GiB`, `56.2 GiB` experts) at full **`262,144` (`256K`) context** with pinned RAM and `--spec 4`.

---

## Hardware and software

- **GPU / VRAM:** 2x AMD Radeon RX 7800 XT (`gfx1101`, 16 GiB each, 32 GiB total) on ASRock X570 Phantom Gaming X (`PCIe 4.0 x8 + x8` direct CPU bifurcation, `14.1 GB/s` engine H2D probe).
- **CPU:** AMD Ryzen 7 5700X (8C/16T, AVX2).
- **RAM:** 64 GB (`60.7 GiB` OS-visible) Kingston DDR4 (`KF3600C18D4/32GX`, running at `2400 MT/s`), `8.0 GiB` swap.
- **Storage:** `INTEL SSDPEKNW010T8` (`953.9 GB` NVMe, PCIe 3.0 x4).
- **OS / ROCm:** Ubuntu `26.04.1 LTS`, kernel `7.0.0-38-generic`, ROCm `10.1` (`HIP 7.16.26385`, `hipBLASLt 100401`).
- **Strata:** `0.1.61` (`v0.2.15`, `c77572806996349b49995e4f98d41b5294d36a21`), variant `rx7900` (`gfx1101`), `gfx1101-hipblaslt-100401.txt` (`43 rows`).

---

## Model and configuration

- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF` — `UD-IQ4_XS` (4 shards, `87.24 GiB`, SHA-256 verified against HF LFS OIDs), packed with `tools/iq_pack.py --compat-bf16` (`1079` tensors, `303` served natively, `1.4 GiB` pack):
  - `00001-of-00004.gguf`: `93ff23a08e5cd7501f24fa16f5b415d5807d0175e04edd3ed6a7a41b601a4eef`
  - `00002-of-00004.gguf`: `13d5dc8b93d039e4cfade5a9af7037f10c97f7ec4e53f75bd2e9ae34aae54aeb`
  - `00003-of-00004.gguf`: `f6b9b5934908a2043732269dd740155d0f921d0763a1dcdd3452246311651f87`
  - `00004-of-00004.gguf`: `5de8b5dbf89409acc39468decafd4a4a95b13c29d88a14b0356107897829ec36`
- **Flags:** `--resident-budget-gib 42 --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --prefill-pipe 1 --pipeline-windows 2 --spec 4 --spec-min-p 0.7 --mtp mtp/rt --max-context 262144 --kv int8 --kv-resident 32768 --conversation-cache-mib 1024 --layer-split auto --vram-reserve-mib 128`
- **Env:** `STRATA_HIPBLASLT_TABLE=on`, `STRATA_HIPBLASLT_TUNING=.../gfx1101-hipblaslt-100401.txt`, `STRATA_HIP_WMMA=1`, `STRATA_GDN_CHUNKED=1`, `STRATA_PF_GEMM=1`, `STRATA_PF_FUSED=1`, `STRATA_SH_STREAM=1`, `STRATA_MTP_STAGE0=1`, `STRATA_NO_ARENA_THP=1`, `STRATA_ROUTE_TAIL_SKIP=0`, `GPU_PINNED_MIN_XFER_SIZE=1048576`.

### Memory residency & auto-sizing note
Unlike `UD-Q4_K_XL` (`104 GiB`, #6202), which required `STRATA_RESIDENT_PIN=0` and `64K` ctx to avoid ROCm `USERPTR` pauses/swap, **`UD-IQ4_XS` fits 100% into `VRAM + pinned RAM` (`file tier: 0.0 GiB`) at full `256K` context**:
- `23.1 GiB` GPU expert cache + **`41.39 GiB` page-locked (pinned) RAM complement** (`3617/3617` borrowed prefill slots kept in RAM, `0` file reads)
- `256K` `int8` KV (`--kv-resident 32768`): `1.55 GiB` pinned RAM
- After `128K` run: **`8.1 GiB` `MemAvailable`** (`1.1 GiB` swap), **`398 / 648 MiB` VRAM free**, **0 driver pauses, 0 major faults, `0.0 MB` expert file reads**
- **Auto-sizing observation:** omitting `--resident-budget-gib` on this `60.7 GiB` OS-visible RAM host caused Strata to attempt pinning the full `54 GiB` expert arena (`MemAvailable` dropped to `730 MiB`, heavy swap), whereas `--resident-budget-gib 42` allocates only the `41.39 GiB` complement and boots cleanly in ~`130s`.

---

## Results (`bench_strata.py`, same method as #6202: `reasoning_effort="none"`, `temperature=0`, `128`-tok cap)

| Scenario | Actual prompt | Reused | Prompt tok/s | Decode tok/s | Engine TTFT | Client TTFT | VRAM hit | PCIe |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Chat (3 runs, median) | 41–49 | 0 | 63.8 | 42.0 (39.6–46.5) | 0.672s | 0.876s | 64.2–86.8% | 3.7–12.5% |
| 4K fresh (2 runs) | 3,994–4,000 | 0 | 850.5 | 49.2 (43.0–55.4) | 4.728s | 6.121s | 89.1–95.4% | 0.8–2.9% |
| 16K fresh (2 runs) | 15,961–16,019 | 0 | 1,509.2 | 54.9 (54.7–55.1) | 10.625s | 11.078s | 97.3–97.5% | 0.3–0.4% |
| 16K cached follow-up | 16,072 | 16,012 | 128.6 | 47.4 | 0.497s | 0.893s | 79.5% | 6.8% |
| 64K fresh | 63,858 | 0 | **1,990.8** | **55.1** | 32.113s | 32.686s | 97.6% | 0.3% |
| 64K cached follow-up | 63,910 | 63,851 | 138.2 | **56.2** | **0.464s** | 1.003s | 87.8% | 3.6% |
| 128K fresh | 127,784 | 0 | **2,007.9** | **48.0** | 63.686s | 64.404s | 97.3% | 0.3% |
| 128K cached follow-up | 127,835 | 127,777 | 138.5 | **56.0** | **0.466s** | 1.058s | 91.3% | 2.2% |

### Additional A/B checks on `UD-IQ4_XS`
1. **Pageable (`STRATA_RESIDENT_PIN=0`, `--resident-budget-gib 34`) vs Pinned (`--resident-budget-gib 42`, `STRATA_PF_FUSED=1`):**
   - Pinned doubled prefill throughput: `16K` `757 → 1,509 tok/s`, `64K` `918 → 1,991 tok/s`, `128K` `911 → 2,008 tok/s` (`0` pauses).
2. **`--spec 3` (MTP drafts=2) vs `--spec 4` (MTP drafts=3) on 10 prompts × 256 tokens (`41 GiB` pinned):**
   - `--spec 4` won `8/10` runs: `16K` cached follow-up median **`46.7`** vs `44.2 tok/s` (`+5.7%`), Python coding `50.3–54.5 tok/s`, for only `+12 MiB` VRAM/GPU (`79.0 MiB` verify buffers).

---

## Summary vs #6202 on the same host

| Quant | Max ctx | 64K fresh prefill | 64K cached decode / TTFT | 128K fresh prefill | 128K cached decode / TTFT |
| --- | ---: | ---: | ---: | ---: | ---: |
| `IQ3_S` (#6202) | `262K` | `2,382 tok/s` | `66.2 tok/s` / `0.429s` | `2,494 tok/s` | `61.6 tok/s` / `0.453s` |
| **`UD-IQ4_XS` (this run)** | **`262K`** | **`1,991 tok/s`** | **`56.2 tok/s` / `0.464s`** | **`2,008 tok/s`** | **`56.0 tok/s` / `0.466s`** |
| `UD-Q4_K_XL` (#6202) | `64K` | `434 tok/s` | `19.2 tok/s` / `1.721s` | — | — |

For `32 GiB VRAM + 64 GB RAM`, **`UD-IQ4_XS` + `--resident-budget-gib 42` (pinned) + `--spec 4` + `--max-context 262144`** is clearly the sweet spot: **4.6× faster prefill**, **~2× faster decode**, and **4× the context** compared to `UD-Q4_K_XL`, while staying close to `IQ3_S` speed with 4-bit quality.

Mehr auf der Site

Links zu Install, Modellen, Releases.