#6526 [Engine] gfx1200 (2x RX 9060 XT, ROCm 10.1): peer-GPU fused WMMA prefill, RDNA4 MMVQ load_int_b2/Q6KTraits, and hipBLASLt fallback notes
open · @Tamanekokko · 0 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
说明
Hi, sharing a few local experiments and measurements from a 2x RX 9060 XT (`gfx1200`) Linux box on ROCm 10.1.0, in case any of the individual pieces are useful upstream. No action or merge is expected from this post — if any piece looks worth turning into a clean opt-in PR (one problem per PR), happy to split and re-verify it; otherwise feel free to treat this as a hardware/measurement note and close it. --- ### Environment & Model - **GPUs**: 2x AMD Radeon RX 9060 XT 16 GB (`gfx1200`, host-bridge / no real P2P) - **Host**: Linux `7.0.0-38-generic`, 60 GiB RAM - **ROCm / Compiler**: `10.1.0` (`AMD clang 24.0.0git`) - **Base commit**: `12d491da4` (`origin/main`, v0.2.17 / engine 0.1.63) - **Model**: `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S` (`--expert-cache auto --peer-device 1 --peer-reserve-mib 512 --prefill 16384 --spec 4 --spec-min-p 0.5 --max-context 262144 --kv int8 --kv-resident 131072 --vision --pcie-frac 0.25`) --- ### 1. Peer-GPU fused WMMA int8 prefill (`STRATA_PF_PEER_FUSED`, `prefill.cpp` + `moe_fused_iq.cu`) **Observation** On a 2x 16 GB `gfx1200` setup (`--peer-device 1`), GPU 1 holds ~8,032 resident experts (vs 4,048 on GPU 0), so a large fraction of routed prompt experts live on the peer GPU. In the stock path, `fz_staged` disables the fused native/WMMA path when `m.pp` is active, leaving peer prefill off `native_w11_kernel`. **Prototype changes** - Added an opt-in path (`STRATA_PF_PEER_FUSED=1`) in `engine/src/prefill/prefill.cpp` allowing GPU 0 and GPU 1 to run fused WMMA int8 (`native_w11_kernel`) concurrently for their respective resident experts, sizing `pp->fr_xq` for the 80-byte/64-elem layout when needed. - On `gfx1200`/`gfx1201` (`engine/src/prefill/moe_fused_iq.cu`), compacted the `IQ2_S` sign LUT in LDS (`SGN=8`, used by `IQ2_S`-quantized routed experts inside the `IQ3_S` pack) and adjusted `nw_capped` / `WLD_NW` for `clang >= 24`. **Measured Prefill (n=1 per size, greedy, fresh server restart per arm, bundled 32-row hipBLASLt table)** Same base commit (`12d491da4`) and ROCm `10.1.0`: stock vs dual-GPU WMMA (`STRATA_PF_PEER_FUSED=1` + `moe_fused_iq.cu` above): - 2,288 fresh tokens: `1356.8` -> `1396.6 tok/s` (`+2.9%`) - 4,494 fresh tokens: `1515.7` -> `1664.1 tok/s` (`+9.8%`) - 9,267 fresh tokens: `1615.7` -> `1682.5 tok/s` (`+4.1%`) - 20,472 fresh tokens: `1640.2` -> `1768.7 tok/s` (`+7.8%`; `gemm gate/up` dropped from `4,207 ms` to `2,793 ms`) - 47,668 fresh tokens: `1714.7` -> `1808.1 tok/s` (`+5.4%`) - Sanity check: 4.5K needle-in-a-haystack (`GAMMA-4928-DELTA`) passed on both arms. (Note: `STRATA_PF_PEER_FUSED` and the `moe_fused_iq.cu` LDS/cap tweak were measured together, not separated.) **Negative results** - Streaming extra non-resident rows to GPU 1 (`STRATA_PF_PEER_STREAM=0.25`) regressed prefill on this host-bridge box because host-staged transfer and combine overhead outweighed the compute gain; keeping GPU 1 to its resident experts only was faster. - `STRATA_Q6_PACKED=1` duplicated ~710 MiB of weights out of the 16 GB VRAM expert-cache budget and net-regressed decode on this 2x 16 GB setup. --- ### 2. RDNA4/HIP `native_mmvq.cu`: `load_int_b2` (`__builtin_memcpy`) + `Q6KTraits` register scales **Observation** `rocprofv3 --kernel-trace` on pure decode showed `native_mmvq_multi_kernel` (~25% of GPU time) and `native_q6_k_mmvq_kernel` (~4.5%) as the largest non-MoE decode consumers. Two HIP-side overheads stood out: 1. `load_int_b2()` manually splits every 32-bit load into two 16-bit loads + shift + OR. Replacing that with `__builtin_memcpy` (preserving `align 2`) lets the AMDGPU backend lower global loads directly where the target supports unaligned 32-bit loads, rather than forcing manual 16-bit packing in source. 2. `q6_q8_dot_sc()` was guarded under `#if !defined(__HIPCC__)`, while `Q6KTraits::W` kept a `const int8_t* scales` pointer instead of hoisting `sc0`/`sc1` into registers during `load()`. **Prototype changes (zero extra VRAM)** - In `load_int_b2`, used `__builtin_memcpy(&value, static_cast<const uint16_t*>(ptr) + 2 * i32, sizeof(int))` (prototyped under `#if defined(__HIPCC__)`, though upstream this should likely be narrowed to `gfx12` unless verified on older AMD targets / ISA). - Exposed `q6_q8_dot_sc` to HIP and hoisted `sc0`/`sc1` in `Q6KTraits::load()` (updating `B6X<Q6KTraits>::wload()` to read `t.sc0`/`t.sc1`). **Measured (isolated kernel microbenchmark on gfx1200, ROCm 10.1.0, `NCOLS=4, ROWS=2`)** - `Q6_K` output head shape `N=248320, K=2560` (both changes combined): `3129.5 us` -> `2593.2 us` (`-17.1%` kernel time / `1.21x` faster; numerical diff vs stock and generated ISA were not separately logged in the microbenchmark). - Scope caveats: this microbenchmark only covers the `Q6_K` output head (~4.5% of decode GPU time); `native_mmvq_multi_kernel` (~25%) was not separately microbenchmarked, and an isolated end-to-end decode A/B for `native_mmvq.cu` alone has not been run yet. --- ### 3. `gfx1200` + ROCm 10.1.0 (`hipBLASLt 100401`) fallback shapes Running with `STRATA_HIPBLASLT_VERBOSE=1` against the bundled 32-row `gfx1200-hipblaslt-100401.txt` showed `Lt fallback; no calibration` on `f16 2560x2560` (at `T=4096..32768`) and `T=2048` buckets (`bf16`/`f16`). Appending those 12 shapes and retuning on ROCm 10.1.0 produced a 44-row table (verified loaded via `prefill gemm: hipBLASLt tuning enabled (44 rows, gfx1200, version 100401)`). *(Note: the prefill runs above used the bundled 32-row table due to a local launcher config override, so the 44-row table's isolated delta has not yet been A/B'd with `tools/hip/bench_prefill.py`.)* --- ### What was NOT tested - CUDA and SYCL builds (only HIP `gfx1200` on Linux was built and run). - Non-`gfx1200` AMD architectures (in particular, whether `__builtin_memcpy` in `load_int_b2` should be guarded to `__gfx1200__`/`__gfx1201__` rather than all `__HIPCC__`, and whether hoisting `sc0`/`sc1` increases VGPR pressure/spills on other targets or `NCOLS`/`ROWS` configs). - Generated ISA inspection for `load_int_b2` and isolated single-factor E2E decode A/B for `native_mmvq.cu`. - `tools/hip/bench_prefill.py` canonical on/off A/B for the 44-row hipBLASLt table. If any single piece (e.g. the `native_mmvq.cu` `Q6KTraits` register-scale / `gfx12` `load_int_b2` cleanup, or a canonical `shapes-canon.json` ROCm 10.1 `gfx1200` hipBLASLt table) is useful, happy to isolate it and open a focused PR with proper single-factor A/B numbers.
本站相关内容
相关页面的快捷入口。