Pull requests / #3476
#3476 [Engine] HIP: verify-window routing for non-native routers (256 experts) on every HIP card
open · @pugolini · 0 コメント · GitHub で見る
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsLinux
本文
## Problem and result The native router is 512 x 10 only, so a 256-expert model (such as Qwen3.8-Flash-Next Coder) fell back to the per-token `moe_route` loop in the verify window on all HIP hardware except `gfx906`: a projection + top-k launch pair per token and layer (~360 small kernel launches in a `--spec 4` step). `moe_route_window` (one multi-column BF16 projection + one top-k per layer, bitwise equivalent to the per-token calls) now runs by default on every HIP card. `STRATA_ROUTE_WIN=0` restores the per-token loop. On CUDA it remains opt-in (`STRATA_ROUTE_WIN=1`) until benchmarked there. ## Related discussion None. ## Approach Enable `route_window_on()` across all HIP cards (and `STRATA_HIP_GFX906`) rather than restricting the fast window router branch to `#if defined(STRATA_HIP_GFX906)`. Also adds profile stamp slot 33 to separate verify window routing from plan/doorbell latency under `STRATA_VERIFY_PROFILE=1`. ## Validation **Passed:** - Tested on `gfx1201` (AMD Radeon AI PRO R9700 32 GB, ROCm 7.10, Linux 7.0), Coder IQ1_M, `--spec 4`, on top of Strata `v0.2.1.1` (`9ae93f7b`): - `route_window_parity`: OK for T in 1..6 (256 experts, top-10 and 1-row gate), bit-identical. - Routing time per verify window: 3.74 ms -> 1.06 ms (GDN), 1.24 ms -> 0.34 ms (QSA). - Short prompts (temp 0): 98.5 -> 108.1 tok/s (+9.7%), 29.5 -> 26.8 ms per step, bitwise identical output hashes. - 45K context prompts (temp 0): 87.1 -> 95.1 tok/s (+9.2%), same output hashes, prefill unchanged. **Failed:** - None observed. **Not run:** - CUDA benchmarking (remains opt-in via `STRATA_ROUTE_WIN=1`). - Intel SYCL / Apple Metal. ## Limits None. Fully bitwise compatible with prior decode output.
関連リンク
インストール・モデル・リリースへのサイト内リンク。