#6264 [Engine] RDNA2 gfx1030: 37% prefill regression persists in 0.1.62 — MoE GEMM gate/up 3.4x slower than 0.1.59

open · @kstoykov · 2 comentários · No GitHub

BenchmarksAMD / HIPModels & quantsLinux

Descrição

<h2><span>Description</span></h2><p><span>I'm opening a new issue as a follow-up to #6067, which has been closed.</span></p><p><span>Unfortunately, the prefill performance regression is </span><strong><span>still present on my AMD Radeon RX 6800 (RDNA2 / gfx1030)</span></strong><span> with Strata 0.2.16 (engine 0.1.62).</span></p><p><span>I've now compared detailed prefill timing logs from </span><strong><span>0.2.13 (engine 0.1.59)</span></strong><span> and </span><strong><span>0.2.16 (engine 0.1.62)</span></strong><span> using the same GPU, model, and engine configuration.</span></p><p><span>The logs clearly show that the majority of the additional processing time comes from MoE GEMM operations, particularly </span><code><span>gemm gate/up</span></code><span>.</span></p><h2><span>Environment</span></h2><ul><li><p><strong><span>GPU:</span></strong><span> AMD Radeon RX 6800 (gfx1030 / RDNA2)</span></p></li><li><p><strong><span>OS:</span></strong><span> Linux</span></p></li><li><p><strong><span>Backend:</span></strong><span> HIP</span></p></li><li><p><strong><span>Model:</span></strong><span> Qwen3.8-Flash-Next-Gyro-S-TQ1_0</span></p></li><li><p><strong><span>Expert pack:</span></strong><span> gyro-gyro-s</span></p></li><li><p><strong><span>Prefill:</span></strong><span> auto (8192-token chunks in both versions)</span></p></li><li><p><strong><span>Resident experts:</span></strong><span> enabled</span></p></li><li><p><strong><span>GPU expert cache:</span></strong><span> 10029 slots (~9.8 GiB)</span></p></li><li><p><strong><span>PCIe bandwidth:</span></strong><span> 27.6 GB/s</span></p></li><li><p><strong><span>Draft:</span></strong><span> DFlash2</span></p></li></ul><p><span>Environment variables:</span></p><pre><code><span>ENV STRATA_DENSE_MMQ="1"
ENV STRATA_ENGINE_BACKEND="hip"
ENV STRATA_HIP_PROMPT_F16="1"
ENV STRATA_MTP_SIDE_STREAM="1"
ENV STRATA_HIPBLASLT_TABLE="on"</span></code></pre><p><span>Both versions confirm that </span><code><span>STRATA_HIP_PROMPT_F16=1</span></code><span> is active.</span></p><h2><span>Performance comparison</span></h2><p><span>The detailed logs contain prefill measurements for </span><strong><span>20,753 tokens</span></strong><span>, with no prompt-cache reuse in either run.</span></p>
Metric | 0.2.13 (engine 0.1.59) | 0.2.16 (engine 0.1.62)
-- | -- | --
Prefill throughput | 867.1 tok/s | 548.2 tok/s
GPU timeline | 23,639 ms | 37,591 ms
Prefill wall time | 23,704 ms | 37,660 ms
Decode throughput | 39.6 tok/s | 39.5 tok/s

<h3><span>Key observation</span></h3><p><span>The total GPU timeline increased by approximately </span><strong><span>13.95 seconds</span></strong><span>.</span></p><p><span>Of that increase:</span></p><ul><li><p><code><span>gemm gate/up</span></code><span> alone accounts for </span><strong><span>10.35 seconds (~74%)</span></strong><span>.</span></p></li><li><p><code><span>gemm down</span></code><span> adds another 1.85 seconds.</span></p></li><li><p><code><span>dequant</span></code><span> adds another 1.86 seconds.</span></p></li><li><p><span>QSA attention and GDN timings remain approximately unchanged.</span></p></li><li><p><code><span>wait copy</span></code><span> is only 353 ms in 0.2.16 (less than 1% of the GPU timeline).</span></p></li></ul><p><span>This strongly suggests that the regression is concentrated in the </span><strong><span>MoE expert compute path</span></strong><span>, rather than attention, CPU processing, or PCIe transfers.</span></p><h2><span>Potential cause</span></h2><p><span>In 0.2.16, the startup log explicitly reports:</span></p><pre><code><span>strata: prompt kernels: every chunk runs 48 of 48 layers on the FP16 path.</span></code></pre><p><span>This message is absent from my 0.2.13 log.</span></p><p><span>Given the changes to the matrix kernels in recent versions, I suspect the newer native FP16/dequantization/GEMM implementation may be substantially slower on </span><code><span>gfx1030</span></code><span> than the previous implementation.</span></p><p><span>However, the timing differences alone do not prove which specific kernel change is responsible.</span></p><p><span>One additional observation: both versions report very low remaining VRAM, so memory pressure is present in both configurations and does not by itself explain the regression.</span></p><h2><span>Expected behavior</span></h2><p><span>Strata 0.2.16 should maintain approximately the same prefill performance as 0.2.13 on RDNA2 GPUs, ideally around 850–900 tok/s for this workload.</span></p><h2><span>Questions</span></h2><ol start="1"><li><p><span>Could the recent changes to the MoE GEMM/dequantization kernels have introduced a performance regression specifically on </span><code><span>gfx1030</span></code><span>?</span></p></li><li><p><span>Is there a runtime or build-time option to restore the previous, faster computation path?</span></p></li><li><p><span>Does the new native implementation require additional optimization or kernel tuning for RDNA2?</span></p></li><li><p><span>Are there any specific profiling options or benchmarks I can run to help isolate the exact kernel responsible?</span></p></li></ol><p><span>I can reproduce the issue consistently and provide additional profiling logs or run targeted benchmarks if needed.</span></p><p><strong><span>Both complete engine logs (0.2.13 and 0.2.16) are available for comparison.</span></strong></p>

[strata-0.2.13.log](https://github.com/user-attachments/files/35075637/strata-0.2.13.log)
[strata-0.2.16.log](https://github.com/user-attachments/files/35075636/strata-0.2.16.log)

No site

Links install, modelos, releases.