Pull requests / #1897
#1897 setup: build the HIP engine with the K-quant MMQ prompt kernels (UD-Q4_K_XL's experts on RDNA2, 633 -> 1,036 tok/s)
open · @elightcap · 0 commentaires · Sur GitHub
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
## Title Issue: relates to #429 and #1278 (no issue of its own) ## Summary `build_engine_hip` in `setup.py` passed `-DSTRATA_PREFILL_MMQ=ON` but not `-DSTRATA_MMQ_KQUANTS=ON`, so a setup-built AMD engine has no MMQ kernels for UD-Q4_K_XL's Q4_K / Q5_K / Q5_1 experts and the prompt path dequantizes every expert to FP16 and multiplies it with hipBLAS. On an RX 6900 XT that was 65-77% of a prompt's GPU time (`STRATA_PREFILL_TIMING=1`: dequant + the two GEMMs), about 5 TFLOPS on the gate/up product of an 8K chunk. The HIP CMake branch already lists the K-quant instances under the option (#1278 registers their test and measured the model on an R9700 built by hand with the flag); this makes setup turn it on, so ggml's RDNA2 MMQ tiles take those formats. The option is compile-time only: packs without K-quant experts run exactly as before. ## What changed - `setup.py`: `-DSTRATA_MMQ_KQUANTS=ON` in the HIP cmake flags, with a comment. The AMD confirmation for UD-Q4_K_XL (#429) now says the model has run on one AMD card (this one) and is untested on the user's, instead of calling the prompt kernels NVIDIA-only; the `nvidia_only` flag and the question itself are unchanged. - `docs/AMD_HIP.md`, RDNA2 section: a bullet with the measurement below, the decode-window profile of the same machine and a per-request `pcie_frac` sweep (0.55 / 0.35 / 0.2 / 0.1 -> 57.2 / 56.2 / 54.2 / 53.3 ms per window, so the smallest share was about 7% faster per window here; the 0.1.42 CUDA rule would pick 0.35). What it leaves alone: CUDA builds, the engine's own MMQ plan (`mmq_plan`, `STRATA_PREFILL_MMQ=0` still gives the FP16 path), `STRATA_DENSE_MMQ` (still opt-in), the gfx11 fused experts. ## Measured RX 6900 XT 16 GB (gfx1030, PCIe 4.0 x16), Ryzen 9 5950X, 128 GB DDR4, Arch Linux, ROCm 7.2.4, engine 0.1.42 built with and without `-DSTRATA_MMQ_KQUANTS=ON` (both with #1683's lend-region fix, which does not touch the kernels), `--resident-budget-gib 71 --prefill auto --kv int8 --kv-resident 32768 --max-context 131072`, `STRATA_HIP_PROMPT_F16=1 STRATA_SH_STREAM=1 STRATA_ROUTE_TAIL_SKIP=7`, one run per cell, greedy, the same prompts: | prompt | without | with K-quant MMQ | |---|---|---| | fresh 7.8K tokens | 633 tok/s (dequant 0.85 s, gate/up 4.5 s, down 1.2 s of 11.3 s) | **1,036 tok/s** (0.39 / 1.15 / 0.52 of 7.0 s) | | fresh 15.8K tokens | 677 tok/s (1.7 / 9.1 / 2.4 of 22.4 s) | **1,104 tok/s** (0.77 / 2.33 / 1.06 of 13.8 s) | | fresh 887 tokens | 4.6 s of GPU time | 3.5 s | | 250-token follow-up chunk on an 8K chat | 2.8 s (dequant 0.44, gate/up 1.24, down 0.59) | 1.8 s (0.24 / 0.16 / 0.08); `wait copy` is then 53% of it | Hidden code words placed mid-document at 8K and 16K were found (one prompt each). Decode is unchanged: 58-68 ms per verify window before and after, 32-38 tok/s. The output is not bit-identical to the FP16 path (int8 MMQ activations), as on CUDA, where this is the path setup builds for the model. After this change the short chunks of an agent's turns are bound by the experts streamed over PCIe; `STRATA_PREFILL_CPU_SHARE=auto` (opt-in on HIP) took the 887-token prompt from 3.75 to 2.65 s and the 250-token chunk from 2.23 to 1.50 s here, but one card is not enough to change that default, so it is only noted. Not tested: another AMD card, Windows HIP, the greedy comparison against llama.cpp, `hip_prefill_mmq_kquant` (tests are off in setup's build; #1278 ran it on an R9700). Measured and written with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HfV9GaWcc3RmPgCvhX2eUf
Sur le site
Liens install, modèles, releases.