#6094 [Engine] gfx103x layer split on PCIe 3.0: STRATA_PREFILL_CPU_SHARE=auto with MAX=16384 reads prompts 1.26-1.54x faster (2x RX 6800 XT)
open · @Bahise06 · 0 comentários · No GitHub
BenchmarksMulti-GPUAMD / HIPModels & quantsDocumentationLinux
Descrição
## Summary This is a data point and a question, not a PR. On 2x RX 6800 XT behind a Gen3-only CPU, the opt-in CPU prompt share gave the largest prompt-read gain of everything I tried, larger than the newer ROCm and the rocBLAS table together. Today it is off by default on HIP layer splits, and the RDNA2 section of `docs/AMD_HIP.md` does not mention it. **Question:** would you take a docs line for it in the RDNA2 section? And would you consider arming it by default on gfx103x layer splits, once someone has a KL number? I can run a KL measurement if you tell me which tool and settings you want. ## Environment 2x RX 6800 XT 16 GB (gfx1030), layer split auto, each at PCIe 3.0 x8 on the CPU's lanes (probe 7.2 / 6.7 GB/s), Ryzen 5 5500 (6 cores, AVX2), 62 GB DDR4, Ubuntu 24.04, kernel 6.8. Engine: the release package `strata-engine-0.1.60-linux-x86_64-hip-rx6000` (v0.2.14), unmodified. Qwen3.8-Flash-Next GSQ-RCO IQ3_S (83.6 GB, so most experts stream), `--prefill auto` (8192-token chunks), `--spec 4 --spec-min-p 0.5 --mtp --kv int8 --kv-resident 32768 --max-context 131072 --pool-workers 5`, `HSA_ENABLE_SDMA=0 STRATA_HIP_PROMPT_F16=1`, the community rocBLAS table and `STRATA_HC_PERSIST=1` in every row. ## Measurements My own script, not the protocol: unique prompts (a random seed at the start, so no prefix reuse), thinking off, medians of 3, one engine start per row. | `STRATA_PREFILL_CPU_SHARE` | `_MAX` | ~4K prompt tok/s | ~16K prompt tok/s | decode tok/s | | --- | --- | ---: | ---: | ---: | | unset (off on this path) | | 440.5 | 812.9 | 50.4 | | `auto` | 8192 | 666.8 | 872.5 | 50.6 | | `auto` | 16384 | **676.9** | **1,022.9** | 52.0 | | `1` | 16384 | 676.4 | 1,029.7 | 51.6 | - With `MAX=8192` the 16K prompt barely moves, because its 8192-token chunks are not below the limit. `MAX` has to exceed the chunk. - Raising the chunk does not help: `--prefill 16384` with `MAX=32768` gave 822 tok/s at 16K, because the prompt path borrows more cache slots. - Standard protocol on the same configuration (results PR #6093): prompt-4k 594.7 and prompt-16k 960.6 tok/s on two cards, against 416.0 / 806.3 for the earlier ROCm 6.4.3 source build without the share. On one card: 584.7 / 782.0 against 394.8 / 613.6. Sanity 12/12 in all runs. - Outside the protocol: a 99,715-token needle prompt read in 66 s with the share, against 83 s without it, and the needle was found. ## Why it pays here Prompt reads on this box are bound by host-to-device bandwidth. Forcing both links to PCIe 2.0 x8 cost 36% of the 4K prompt rate and 31% at 16K, with decode unchanged. Each expert the CPU computes is one that does not cross a slow link. I expect the gain to shrink on PCIe 4.0 machines (#5997's box reads at 14.1 GB/s), but I have not measured that. ## Not measured - KL or any quality metric beyond the 12 sanity checks. - One card with the share alone (the one-card standard run above includes the newer package and the table too). - Interaction with `--batch` (I run one slot).
No site
Links install, modelos, releases.