Pull requests / #3085

#3085 Enable IQ prefill path on sm_75 (RTX 20-series / Turing)

open · @dd777mine · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

<!-- If Applicable, reference the GitHub issue -->
Issue: N/A

## Summary
<!-- Quick Summary of changes -->
On Turing (`sm_75`, e.g. RTX 2060), `moe_fused_iq.cu` was bailing out early under the assumption that codebook formats do not fit 64 KB of shared memory at 4 warps a row. However, with `ASTAGES_DEV=2`, `IQ3_S` blocks require ~58.8 KB (cleanly fitting the 64 KB ceiling), and `IQ2_S` / `IQ2_XS` fit at 2 warps (~43.5 KB).

This PR adjusts the sm_75 gate so Turing GPUs can use the native fused MoE prefill path instead of falling back to CPU dequantization and host synchronization.

## What changed
<!-- Specifics on files changed, and what changes were made there -->
One file: `src/prefill/moe_fused_iq.cu`

* `setup_one_turing()`: registers `IQ2_S` / `IQ2_XS` at `WW=2` only; all other formats keep `WW=4` + `WW=2` (matching sm_80+).
* `dev_info()`: runs `setup_one_turing` for the codebook formats on Turing instead of restricting to `Q2_0`.
* `native_supported()`: removes the sm_75-only `Q2_0` restriction and checks `d.t75` directly so either `STRATA_PF_FUSED=1` or `STRATA_PF_FUSED_SM75=1` activates the path.
* `experts_native()`: pins `ww=2` for `IQ2_S` / `IQ2_XS` on Turing to match what was registered.
* Fully opt-in: activates with either `STRATA_PF_FUSED=1` or `STRATA_PF_FUSED_SM75=1`. Non-Turing (`sm_80+`) and HIP builds remain byte-identical.

## Extra Notes
<!-- Any extra notes, delete if there are none -->
### Numbers (RTX 2060 12 GB, Qwen3.8-Flash-Next IQ3_S)
Tested on ~5.8K prompt tokens (clean v0.1.45 build, only this file changed):

* **Time to first token (prompt latency):** 14.1 s → **12.3 s** (-1.8 s faster)
* **Bulk prefill speed:** 509 tok/s → **606 tok/s** (+19.2%)
* **Decode speed:** identical (zero regression)

### Context & Transparency
* I run an RTX 2060 12 GB at home where prefill was the main bottleneck. I worked on tracing the gate with an AI coding agent as a second pair of eyes, and verified everything by building and running it on real hardware. I'm not a CUDA expert, and would rather say that up front.
* In my personal fork I have a few other experimental tweaks (enabling BF16 tensor cores in `gemm.cu` and GDN vectorization), which push prefill speed even further to **~760 tok/s** (10.1 s latency, **+50%** total speedup). None of that is in this PR — I wanted to keep this strictly scoped to one file and one clear fix per contribution guidelines, but happy to share or open separate PRs if interested.
* If any of the setup checks fail on other hardware, `d.ok` stays false and the engine safely falls back to the existing non-native path.
* Happy to rename things, split this down, or close if you prefer a different approach for Turing. Feedback welcome either way!

Mehr auf der Site

Links zu Install, Modellen, Releases.