Pull requests / #3085
#3085 Enable IQ prefill path on sm_75 (RTX 20-series / Turing)
open · @dd777mine · 0 commentaires · Sur GitHub
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quants
Description
<!-- If Applicable, reference the GitHub issue --> Issue: N/A ## Summary <!-- Quick Summary of changes --> On Turing (`sm_75`, e.g. RTX 2060), `moe_fused_iq.cu` was bailing out early under the assumption that codebook formats do not fit 64 KB of shared memory at 4 warps a row. However, with `ASTAGES_DEV=2`, `IQ3_S` blocks require ~58.8 KB (cleanly fitting the 64 KB ceiling), and `IQ2_S` / `IQ2_XS` fit at 2 warps (~43.5 KB). This PR adjusts the sm_75 gate so Turing GPUs can use the native fused MoE prefill path instead of falling back to CPU dequantization and host synchronization. ## What changed <!-- Specifics on files changed, and what changes were made there --> One file: `src/prefill/moe_fused_iq.cu` * `setup_one_turing()`: registers `IQ2_S` / `IQ2_XS` at `WW=2` only; all other formats keep `WW=4` + `WW=2` (matching sm_80+). * `dev_info()`: runs `setup_one_turing` for the codebook formats on Turing instead of restricting to `Q2_0`. * `native_supported()`: removes the sm_75-only `Q2_0` restriction and checks `d.t75` directly so either `STRATA_PF_FUSED=1` or `STRATA_PF_FUSED_SM75=1` activates the path. * `experts_native()`: pins `ww=2` for `IQ2_S` / `IQ2_XS` on Turing to match what was registered. * Fully opt-in: activates with either `STRATA_PF_FUSED=1` or `STRATA_PF_FUSED_SM75=1`. Non-Turing (`sm_80+`) and HIP builds remain byte-identical. ## Extra Notes <!-- Any extra notes, delete if there are none --> ### Numbers (RTX 2060 12 GB, Qwen3.8-Flash-Next IQ3_S) Tested on ~5.8K prompt tokens (clean v0.1.45 build, only this file changed): * **Time to first token (prompt latency):** 14.1 s → **12.3 s** (-1.8 s faster) * **Bulk prefill speed:** 509 tok/s → **606 tok/s** (+19.2%) * **Decode speed:** identical (zero regression) ### Context & Transparency * I run an RTX 2060 12 GB at home where prefill was the main bottleneck. I worked on tracing the gate with an AI coding agent as a second pair of eyes, and verified everything by building and running it on real hardware. I'm not a CUDA expert, and would rather say that up front. * In my personal fork I have a few other experimental tweaks (enabling BF16 tensor cores in `gemm.cu` and GDN vectorization), which push prefill speed even further to **~760 tok/s** (10.1 s latency, **+50%** total speedup). None of that is in this PR — I wanted to keep this strictly scoped to one file and one clear fix per contribution guidelines, but happy to share or open separate PRs if interested. * If any of the setup checks fail on other hardware, `d.ok` stays false and the engine safely falls back to the existing non-native path. * Happy to rename things, split this down, or close if you prefer a different approach for Turing. Feedback welcome either way!
Sur le site
Liens install, modèles, releases.