Pull requests / #3283
#3283 sm_70: keep the CUDA layout for the IQ4_NL down projection under Volta mode 8
open · @aleesposito85 · 0 comentários · No GitHub
Descrição
sm_70: keep the CUDA layout for the IQ4_NL down projection under Volta mode 8 ## What On Volta (sm_70) with the mode-8 expert layout (`STRATA_SM70_TABLE=1`), `native_expert_grouped` routes the down projection through `native_down_lds_kernel` for both `IQ4_NL` (20) and `Q2_0` (42). `native_expert_bench` on an IQ3_S pack's `iq2_s`/`iq4_nl` layers shows that choice is a loss for IQ4_NL — the CUDA layout is faster — while mode 8 keeps its win on gate/up. This mirrors the existing `sm75_down_em()`: a new `sm70_down_em(em0, d_type)` returns mode 0 for `d_type == 20` on compute capability 7.0. `Q2_0` (42), every other type and every other arch keep their mode, and `STRATA_EXP_MODE` still wins when set. ## Measured Tesla V100-SXM2-32GB, CUDA 12.8, `native_expert_bench`, real rows of the IQ3_S pack, 10 groups, medians of 300. `all` is the per-layer expert call (gate/up + fused SwiGLU + q8_1 + down). | layer | types | T=1 before | T=1 after | T=4 before | T=4 after | | ---: | --- | ---: | ---: | ---: | ---: | | 3 | iq2_s / iq4_nl | 63.9 us | **59.1 us (-7.5%)** | 94.7 us | **87.1 us (-8.0%)** | | 40 | iq3_s / iq4_nl | 59.1 us | **54.0 us (-8.6%)** | 87.7 us | **76.6 us (-12.7%)** | | 10 | iq2_s / q2_0 | 54.0 us | 54.3 us | 84.2 us | 84.4 us | | 20 | iq3_xxs / q2_0 | 46.7 us | 46.7 us | 75.9 us | 75.8 us | The down phase alone on the IQ4_NL layers: 25.8 -> 20.7 us (T=1), 39.2 -> 28.3 us (T=4); gate/up is untouched (36.3 us at T=1, 55.0 us at T=4, the mode-8 win that stays). Every row is **bitwise equal** to the reference mode (the bench checks every cell), so this changes speed only. ## Tests - `native_expert_bench` … bitwise equal on all rows above. - Builds with `-DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON` (CUDA 12.8).
No site
Links install, modelos, releases.