Pull requests / #3175

#3175 [Benchmark] 2x MI50 16 GB (gfx906): IQ3_XXS at 128K, --pipeline-windows 2 gives +36% decode

open · @romeo5353535353-ui · 0 comentarios · En GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Descripción

## Problem and result

There is no measured report for the gfx906 build with the original IQ3_XXS at the
full 131,072-token context, and `docs/AMD_HIP.md` does not mention the two flags
that matter most on a two-card gfx906 split.

This adds a community report: 2x Instinct MI50 16 GB + Ryzen 7 5700X, Strata
v0.1.43.1, IQ3_XXS at 131,072 tokens. The two findings worth a line in the docs:

- `--pipeline-windows 2`: decode **49.2 -> 66.9 tok/s (+36%)**, prompt rate
  unchanged (353 tok/s).
- `--trim-stage-weights` (with the explicit `--layer-split 27`): prompt reading
  **353 -> 371 tok/s (+5%)**, decode +2%.

Also measured and reported so others do not repeat it: `--batch` gives no
aggregate gain for one interactive user (per-request decode collapses), RAM
2400 -> 3600 MT/s is +2%, and `--kv-resident` from 0 to 65536 is neutral both at
1.6K and at 31K prompt tokens.

## Related discussion

https://github.com/Niko1221/Strata/issues/3173

## Approach

Documentation and data only:

- `bench/results/2026-10-19-community-2x-mi50-5700x/README.md` - the report,
  following the format of the other community reports;
- `bench/results/2026-10-19-community-2x-mi50-5700x/results.json` - every run,
  not only the medians;
- one bullet in `docs/COMMUNITY_BENCHMARKS.md`.

No code, no defaults, no build changes.

## Validation

**Passed:** 12 configurations, each a full container restart plus one discarded
warm-up and 7 long + 5 short measured requests. Engine v0.1.43.1 (`a0116b2`),
Ubuntu 24.04.5, kernel 6.8.0-147, ROCm from `mixa3607/rocm-gfx906:7.14-complete`,
`-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906`. About 1.5 h of
continuous load across the runs: no engine errors, no faults, `dmesg` clean, GPU
junction 45-49 C, 200 W cap per card.

**Failed:** none observed.

**Not run:** no source changes, so no CUDA / HIP / SYCL / Metal builds were made;
a single machine; the sweep prompt is 1.6K tokens with one 31K probe, not the
4K/32K/128K ladder of the harness.

## Limits

One machine, one VBIOS (the cards report `1002:66af`), MTP on, 200 W per card.
The report is measured on 0.1.43.1 because 0.1.44+ does not build for gfx906 on
this machine (#3172).

En el sitio

Enlaces a install, modelos, releases.