Pull requests / #3175
#3175 [Benchmark] 2x MI50 16 GB (gfx906): IQ3_XXS at 128K, --pipeline-windows 2 gives +36% decode
open · @romeo5353535353-ui · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
## Problem and result There is no measured report for the gfx906 build with the original IQ3_XXS at the full 131,072-token context, and `docs/AMD_HIP.md` does not mention the two flags that matter most on a two-card gfx906 split. This adds a community report: 2x Instinct MI50 16 GB + Ryzen 7 5700X, Strata v0.1.43.1, IQ3_XXS at 131,072 tokens. The two findings worth a line in the docs: - `--pipeline-windows 2`: decode **49.2 -> 66.9 tok/s (+36%)**, prompt rate unchanged (353 tok/s). - `--trim-stage-weights` (with the explicit `--layer-split 27`): prompt reading **353 -> 371 tok/s (+5%)**, decode +2%. Also measured and reported so others do not repeat it: `--batch` gives no aggregate gain for one interactive user (per-request decode collapses), RAM 2400 -> 3600 MT/s is +2%, and `--kv-resident` from 0 to 65536 is neutral both at 1.6K and at 31K prompt tokens. ## Related discussion https://github.com/Niko1221/Strata/issues/3173 ## Approach Documentation and data only: - `bench/results/2026-10-19-community-2x-mi50-5700x/README.md` - the report, following the format of the other community reports; - `bench/results/2026-10-19-community-2x-mi50-5700x/results.json` - every run, not only the medians; - one bullet in `docs/COMMUNITY_BENCHMARKS.md`. No code, no defaults, no build changes. ## Validation **Passed:** 12 configurations, each a full container restart plus one discarded warm-up and 7 long + 5 short measured requests. Engine v0.1.43.1 (`a0116b2`), Ubuntu 24.04.5, kernel 6.8.0-147, ROCm from `mixa3607/rocm-gfx906:7.14-complete`, `-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906`. About 1.5 h of continuous load across the runs: no engine errors, no faults, `dmesg` clean, GPU junction 45-49 C, 200 W cap per card. **Failed:** none observed. **Not run:** no source changes, so no CUDA / HIP / SYCL / Metal builds were made; a single machine; the sweep prompt is 1.6K tokens with one 31K probe, not the 4K/32K/128K ladder of the harness. ## Limits One machine, one VBIOS (the cards report `1002:66af`), MTP on, 200 W per card. The report is measured on 0.1.43.1 because 0.1.44+ does not build for gfx906 on this machine (#3172).
Mehr auf der Site
Links zu Install, Modellen, Releases.