Pull requests / #3283

#3283 sm_70: keep the CUDA layout for the IQ4_NL down projection under Volta mode 8

open · @aleesposito85 · 0 comentarios · En GitHub

needs-format

Descripción

sm_70: keep the CUDA layout for the IQ4_NL down projection under Volta mode 8

## What

On Volta (sm_70) with the mode-8 expert layout (`STRATA_SM70_TABLE=1`), `native_expert_grouped`
routes the down projection through `native_down_lds_kernel` for both `IQ4_NL` (20) and `Q2_0` (42).
`native_expert_bench` on an IQ3_S pack's `iq2_s`/`iq4_nl` layers shows that choice is a loss for
IQ4_NL — the CUDA layout is faster — while mode 8 keeps its win on gate/up.

This mirrors the existing `sm75_down_em()`: a new `sm70_down_em(em0, d_type)` returns mode 0 for
`d_type == 20` on compute capability 7.0.  `Q2_0` (42), every other type and every other arch keep
their mode, and `STRATA_EXP_MODE` still wins when set.

## Measured

Tesla V100-SXM2-32GB, CUDA 12.8, `native_expert_bench`, real rows of the IQ3_S pack, 10 groups,
medians of 300.  `all` is the per-layer expert call (gate/up + fused SwiGLU + q8_1 + down).

| layer | types | T=1 before | T=1 after | T=4 before | T=4 after |
| ---: | --- | ---: | ---: | ---: | ---: |
| 3 | iq2_s / iq4_nl | 63.9 us | **59.1 us (-7.5%)** | 94.7 us | **87.1 us (-8.0%)** |
| 40 | iq3_s / iq4_nl | 59.1 us | **54.0 us (-8.6%)** | 87.7 us | **76.6 us (-12.7%)** |
| 10 | iq2_s / q2_0 | 54.0 us | 54.3 us | 84.2 us | 84.4 us |
| 20 | iq3_xxs / q2_0 | 46.7 us | 46.7 us | 75.9 us | 75.8 us |

The down phase alone on the IQ4_NL layers: 25.8 -> 20.7 us (T=1), 39.2 -> 28.3 us (T=4); gate/up is
untouched (36.3 us at T=1, 55.0 us at T=4, the mode-8 win that stays).

Every row is **bitwise equal** to the reference mode (the bench checks every cell), so this changes
speed only.

## Tests

- `native_expert_bench` … bitwise equal on all rows above.
- Builds with `-DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON` (CUDA 12.8).

En el sitio

Enlaces a install, modelos, releases.