Pull requests / #3104
#3104 Extend native-IQ prefill dispatch to SM75/Turing
open · @c3c · 0 コメント · GitHub で見る
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
本文
This the rebased version of https://github.com/Niko1221/Strata/pull/1581 for the native-IQ prefill part of the PR.
## Summary
This PR extends the existing native fused-IQ expert prefill path to SM75/Turing.
It uses a qualified 128-row Turing shape for supported IQ formats and leaves
unsupported pairs on the existing MMQ path. The upstream SM80+ native-IQ path
is unchanged.
The default path is unchanged when the opt-in is absent. Routing IDs, selected
experts, route weights and scatter order are unchanged.
## What changed
- Added SM75 native-IQ dispatch behind:
```json
{
"STRATA_PF_FUSED": "1",
"STRATA_PF_FUSED_SM75": "1",
"STRATA_PF_FUSED_NATIVE_SM75": "1"
}
```
- Supported SM75 gate/up formats: IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S and
IQ4_XS.
- Supported down formats for the native combinations: Q2_0 and IQ4_NL.
- Unsupported formats continue through MMQ.
- Updated the native-IQ kernel admission and parity test coverage.
- Touched `src/prefill/moe_fused_iq.cu`,
`include/strata/prefill/moe_fused_iq.hpp` and
`tests/cuda/prefill_fused_iq_test.cpp`.
- Updated `docs/DETAILS.md` with the opt-in and format scope.
`STRATA_PF_FUSED` and `STRATA_PF_FUSED_SM75` were already upstream switches.
This PR adds `STRATA_PF_FUSED_NATIVE_SM75` as a separate Turing native-IQ A/B
gate; it extends the existing native-IQ coverage rather than replacing it.
Existing users who set only `STRATA_PF_FUSED_SM75` retain the old Q2_0 behavior
and do not dispatch the new IQ kernels. On SM80+ the existing native-IQ gate and
dispatch are unchanged.
| Variable | Status in this PR |
| --- | --- |
| `STRATA_PF_FUSED` | Existing upstream opt-in; still required |
| `STRATA_PF_FUSED_SM75` | Existing upstream Turing Q2_0 opt-in; still required |
| `STRATA_PF_FUSED_NATIVE` | Existing upstream A/B disable switch; not required and not changed |
| `STRATA_PF_FUSED_NATIVE_SM75` | New native-IQ SM75 A/B gate |
## Extra Notes
### Test system and method
- GPU: NVIDIA GeForce RTX 2080 Super Max-Q, 8 GiB, 105 W, compute capability
7.5.
- CPU/RAM: Intel i7-10875H, 64 GiB RAM.
- Link: PCIe 3.0 x16; measured host-to-device bandwidth approximately
12.2 GB/s.
- Platform/build: Windows, CUDA 13.4, MSVC Release build with SM75 SASS.
- Engine: current upstream main at v0.1.45 (`477ac13c`).
- Model: Qwen3.8-Flash-Next IQ3_XXS, native GGUF plus IQ3_XXS expert pack.
- Runtime: 65,536-token context, Q4_0 KV, 24,576 resident KV cells, MTP
enabled with `--spec 4`, and the experimental speed projection held constant.
- Benchmark request: 31,258 fresh prompt tokens and a 512-token completion
cap. Runs were serialized with a fresh engine; PP/TG are the engine-reported
timing-line rates.
### Correctness and dispatch safety
The SM75 kernel admission checks the actual CUDA compute capability and
occupancy. Setting the SM75 flag does not make SM70, HIP or another unsupported
device execute Turing kernels. Each supported format pair is admitted
independently; an unsupported or non-fitting pair falls back to MMQ. The
change is prompt-prefill-only and does not alter decode allocations.
The native flag is an admission request, not a forced override. On this 8-GiB
card, normal startup keeps the verifier, native output head and MTP resources
resident, leaving too little temporary VRAM for a useful native fused shape.
The planner then falls back to MMQ. The separate [VRAM loans](https://github.com/Niko1221/Strata/pull/3105) defers those
decode-only resources and makes the native shape fit; the loan is not an
algorithmic prerequisite on GPUs with more free VRAM.
### Measurements
The native-IQ parity matrix passed for every admitted pair. Fused-versus-MMQ
relative RMS was approximately `3.5e-5` to `2.6e-4`; the pre-existing Q2_0
pair was approximately `4.0e-4`. The Release build produced both `strata.exe`
and `prefill_fused_iq_test.exe` successfully.
The strongest end-to-end screen used this PR together with the separate
request-scoped VRAM-loan opt-in, see https://github.com/Niko1221/Strata/pull/3105:
| Configuration | Prompt plan | PP | TG |
| --- | ---: | ---: | ---: |
| Current-main control; native path not admitted | 512 / 8 | 294.6 | 30.0 |
| [VRAM loans](https://github.com/Niko1221/Strata/pull/3105) only | 5,376 / 227 | 780.6 | 29.7 |
| [VRAM loans](https://github.com/Niko1221/Strata/pull/3105) + SM75 native-IQ | 4,864 / 512 | **870.1** | 30.7 |
The native-IQ-plus-loan run is **11.5% faster in PP** than the loan-only run in this screen.
This is a practical end-to-end comparison, not a kernel-isolated percentage: enabling native-IQ changes both
supported expert execution and the admitted prompt chunk/ring geometry.
No standalone native-IQ PP percentage is claimed because the native-only
configuration did not admit the native planner on this 8-GiB card.
This path is independent of the upstream `--prefill-head-alias` option and can
be used with it. The separate VRAM-loan PR intentionally rejects
`--prefill-head-alias` because both features manage the native output-head
allocation at the prompt/decode boundary.関連リンク
インストール・モデル・リリースへの站内リンク。