Pull requests / #386
#386 tools/hip: gfx1201 hipBLASLt table for ROCm 7.2.4 (hipBLASLt 1.2.2), ~1.8x prompt speed on the R9700
closed · @abhinand5 · 0 comments · View on GitHub
BenchmarksSetup & installAMD / HIPModels & quantsDocumentation
Description
The RDNA4 notes say a gfx1201 hipBLASLt table only moved prompt speed by 0-3%, so none ships. That matches hipBLASLt 1.4.1, but on a system ROCm 7.2.4 (hipBLASLt 1.2.2) the plain hipBLAS path is much slower, and a table almost doubles prompt speed. This adds one for that version. `tools/hip/gfx1201-hipblaslt-100202.txt` was calibrated on my R9700 with `tune_hipblaslt` at the engine's shapes. It covers 24 of the 26 shapes in the gfx1100 table; the other two fall back to hipBLAS as usual. Since setup only uses a table whose arch and version match exactly, nothing changes for anyone on TheRock wheels or ROCm 7.14. **Numbers** (R9700 32 GB, Ryzen 7 9800X3D, 30 GB single-channel DDR5, full GSQ-RCO IQ3_XXS native pack with `--mmap-experts`, 15,602 cached slots, `--kv int8 --kv-resident 32768 --max-context 131072`, `STRATA_STAGER_THREADS=8 STRATA_STAGER_RING=32`, engine 0.1.31.) The only difference between the arms is `STRATA_HIPBLASLT_TUNING`. I alternated the arms across two server starts and took the median of 4 warm runs each: | | no table | table | |---|---|---| | fresh 32K prompt | 638 tok/s | 1,177 tok/s | | fresh 7K prompt | 656 tok/s | 1,164 tok/s | | decode after the prompt | 44.2-44.6 tok/s | 44.2-44.4 tok/s | `hip_prefill_hipblaslt_gemm`, which normally skips for lack of a table, passes with this one on all four cases (bf16 and f16, relative L2 at most 2.6e-6). I also updated the hipBLASLt bullet in the RDNA4 section of `docs/AMD_HIP.md`, so it says the 1.4.1 result doesn't carry over to 1.2.2. My guess is that your 0-3% and my +80% are both right, and the gap is the hipBLASLt version rather than the model. I haven't checked that on 1.4.1 myself, though. --- *This PR was drafted with AI assistance (Claude), and tested on my machine.* 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.