Issues / #2461
#2461 gfx906 + 100% expert residency: `STRATA_SPLIT_OWN=1` costs decode, `STRATA_PREFILL_HELP=1` does nothing
open · @kietq8684-design · 0 comments · View on GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
### Summary On a gfx906 box whose experts are 100% resident in VRAM, two of the opt-in switches documented in `MULTI_GPU.md` do not pay off, and one of them is actively negative. The published numbers for both were taken on other architectures (`SPLIT_OWN`: R9700 + RX 9070 XT; `PREFILL_HELP`: 2x RTX 3090) and neither section states that residency changes the outcome. Reporting so the sections can carry a precondition. ### Environment - Radeon VII 16 GB (`0x66af`) + Radeon Pro VII / Instinct MI50 16 GB (`0x66a1`), Ryzen 7 3700X (AVX2, no AVX-512), 62 GB RAM, kernel 6.8.0-146-p2p - HIP 7.14 from `mixa3607/rocm-gfx906:7.14-complete`; engine v0.1.42 (`61b3fb5`), gfx906 build - Coder IQ1_M (`--native` pack), `--layer-split 27`, `--prefill 4096`, `--kv int8 --kv-resident 32768`, `--max-context 32768`, `--pcie-frac 0`, `--pipeline-windows 2`, `--ple-row-cache 8388608`, `--spec 4 --spec-min-p 0.7` - `STRATA_ARENA_MMAP=1 STRATA_STAGE_TRIM=1 STRATA_GFX906_ATTN_QUERY_SWIZZLE=1 STRATA_GFX906_ATTN_REDUCE12=1` - Startup: 6,889 + 5,376 = **12,265 experts, 100% resident**, prompt path borrows 1,040 + 937 slots, ring 96 slots ### Method Interleaved A/B (`A1 B1 A2 B2 A3 B3`), one independent restart per arm, 3 runs per arm per prompt length, n=9/arm/length, 4K = 4,589 prompt tokens, 16K = 18,010 prompt tokens, `temperature=0`, unique nonce per request. Interleaving is necessary on this box: two runs of the *same* binary 1.5 h apart differ by ~2%. Prefill is kept as a control column (it should not move). Reported values are medians. ### `STRATA_SPLIT_OWN=1` — prefill barely moves, decode drops Structural evidence: the expert cache falls from **6,889 to 5,353 slots (−22%, −2.82 GiB)**, because each card now keeps its own ~3,264 MiB prompt buffers. | | default | `SPLIT_OWN=1` | delta | |---|---|---|---| | prefill 4K (tok/s) | 469.5 | 479.9 | +2.2% | | prefill 16K (tok/s) | 735.2 | 741.1 | +0.8% | | **decode 4K (tok/s)** | **75.50** | **72.80** | **−3.6%** | | **decode 16K (tok/s)** | **70.10** | **66.40** | **−5.3%** | `MULTI_GPU.md:80` reports 2K 1,450 / 16K 2,227 tok/s there with "decode unchanged". On gfx906 the prefill gain is +0.8–2.2% instead of ~+14%, and the cache it costs is what decode needs. Decode is the primary metric for this deployment, so we leave it off. Suggestion: state in that section that the flag trades expert cache for prompt buffers, and that on a card whose cache is the binding constraint the decode cost can exceed the prefill gain. ### `STRATA_PREFILL_HELP=1` — no effect at all 1.5K prefill **415.1 vs 419.7 tok/s (−1.1%)**, 3K **460.1 vs 460.0 (flat)**, n=3/arm. The decode deltas (+4% / −4.5%) contradict each other and the flag only touches the prompt path, so they are noise. Mechanism as we understand it: the flag hands "a share of its **streamed** experts" to the idle card (`MULTI_GPU.md:85`). With `STRATA_STAGE_TRIM=1` every expert is resident, so the prompt path streams essentially nothing — the work it is meant to offload does not exist. Suggestion: add "requires non-resident (streamed) experts" as a precondition to that section, next to the existing MMQ/`--peer-device` notes. ### For completeness, `STRATA_GR_SPLIT=1` `AMD_HIP.md:494` gives it "~3% per window". End to end we measure 4K +0.8% / 16K −1.6% (prefill flat, n=9) — within noise. That may just be a per-window vs end-to-end distinction rather than a disagreement, so flagging it only as a data point. ### Not a claim This says nothing about the flags on the architectures they were measured on; it is one gfx906 box with a fully resident expert cache.
Related on this site
Quick links to related pages.