Pull requests / #2799

#2799 HIP: verify-window routing for non-native routers (256 experts) on every HIP card

open · @pugolini · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsLinux

Descripción

## Summary

The native router is 512 × 10 only. A 256-expert model (the Coder) therefore takes the per-token `moe_route` loop inside the verify window on every card except gfx906: one projection + top-k launch pair per token and layer, about 360 small launches per `--spec 4` step. `moe_route_window` (one multi-column BF16 projection + one top-k per layer, bitwise equal to the per-token calls per `route_window_parity`) was compiled for gfx906 only.

This PR runs it on every HIP card by default. `STRATA_ROUTE_WIN=0` restores the per-token loop. On CUDA it is opt-in (`STRATA_ROUTE_WIN=1`), because I have not measured it there.

**Relation to #238:** this does not touch the native router or its 512-only guard. The Coder keeps the generic router's arithmetic; only the number of launches changes, and the output stays byte-identical (below).

## Commits

1. **Verify profile + parity test:** a profile-only stamp after the routing splits slot 17 into `router (proj+topk)` and `plan+bell`, so a slow router shows on its own line (it writes nothing unless `STRATA_VERIFY_PROFILE` or the #649 trace is on). `route_window_parity` now also checks T = 1 and 5 (the `--spec 4` verifier runs every T from 1 to 5).
2. **HIP:** the `STRATA_HIP_GFX906` guard around the `moe_route_window` branch becomes `route_window_on()`.

## Measurements

Radeon AI PRO R9700 32 GB (gfx1201, ROCm SDK 7.10 wheels), i7-14700, 32 GB RAM, Linux. Coder IQ1_M, `--spec 4`, MTP, resident experts (100 % VRAM hit rate). Engine built from this branch on main `296b59d2`, default gfx12 settings (so `STRATA_HC_Q8` was on in both arms). A = `STRATA_ROUTE_WIN=0` (current main behaviour), B = default.

- `route_window_parity`: **OK**, T = 1..6, 256 experts top-10 and the 1-row gate, bit-identical.
- Routing (projection + top-k) per verify window, `STRATA_VERIFY_PROFILE=1`:
  - GDN layers: 3.74 → 1.06 ms
  - QSA layers: 1.24 → 0.34 ms

| Benchmark (temp 0) | A tok/s | B tok/s | ms/step A → B | Output |
|---|---|---|---|---|
| 5 short code/text prompts, 600 max tokens | 98.5 | **108.1 (+9.7 %)** | 29.5 → 26.8 | same text, 5/5 |
| 3 prompts at 45K context, 500 max tokens | 87.1 | **95.1 (+9.2 %)** | 30.8 → 28.3 | same text, 3/3 |

Prefill is unchanged. Acceptance is unchanged (74.9 % / 74.8 %).

## Compatibility with open PRs

This branch merges cleanly with #2492 (gfx12 grid padding) and #2476 (no file lookahead when experts are resident). I run all three together on the same R9700: 13/13 served answers byte-identical to main (5 short, 5 Spanish, 3 at 45K context), with the measurements posted on those PRs.

## Not tested

- CUDA: the branch is guarded as opt-in there.
- gfx906: unchanged by default (it already took this path).
- Other RDNA cards: the path is architecture-independent and parity-checked, but I have only measured gfx1201.

En el sitio

Enlaces a install, modelos, releases.