Pull requests / #3476

#3476 [Engine] HIP: verify-window routing for non-native routers (256 experts) on every HIP card

open · @pugolini · 0 comments · View on GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

## Problem and result
The native router is 512 x 10 only, so a 256-expert model (such as Qwen3.8-Flash-Next Coder) fell back to the per-token `moe_route` loop in the verify window on all HIP hardware except `gfx906`: a projection + top-k launch pair per token and layer (~360 small kernel launches in a `--spec 4` step).

`moe_route_window` (one multi-column BF16 projection + one top-k per layer, bitwise equivalent to the per-token calls) now runs by default on every HIP card. 
`STRATA_ROUTE_WIN=0` restores the per-token loop. On CUDA it remains opt-in (`STRATA_ROUTE_WIN=1`) until benchmarked there.

## Related discussion
None.

## Approach
Enable `route_window_on()` across all HIP cards (and `STRATA_HIP_GFX906`) rather than restricting the fast window router branch to `#if defined(STRATA_HIP_GFX906)`. Also adds profile stamp slot 33 to separate verify window routing from plan/doorbell latency under `STRATA_VERIFY_PROFILE=1`.

## Validation
**Passed:**
- Tested on `gfx1201` (AMD Radeon AI PRO R9700 32 GB, ROCm 7.10, Linux 7.0), Coder IQ1_M, `--spec 4`, on top of Strata `v0.2.1.1` (`9ae93f7b`):
  - `route_window_parity`: OK for T in 1..6 (256 experts, top-10 and 1-row gate), bit-identical.
  - Routing time per verify window: 3.74 ms -> 1.06 ms (GDN), 1.24 ms -> 0.34 ms (QSA).
  - Short prompts (temp 0): 98.5 -> 108.1 tok/s (+9.7%), 29.5 -> 26.8 ms per step, bitwise identical output hashes.
  - 45K context prompts (temp 0): 87.1 -> 95.1 tok/s (+9.2%), same output hashes, prefill unchanged.

**Failed:**
- None observed.

**Not run:**
- CUDA benchmarking (remains opt-in via `STRATA_ROUTE_WIN=1`).
- Intel SYCL / Apple Metal.

## Limits
None. Fully bitwise compatible with prior decode output.

Related on this site

Quick links to related pages.