#6604 [Engine] sm_70: unrolled Q8_0 GEMV for 2-4 columns (STRATA_Q8_V70, opt-in, bitwise)
open · @danexpo · 0 评论 · 去 GitHub 看
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
说明
## Problem and result
On sm_70 the default multi-column Q8_0 kernel walks a row's blocks in a loop whose trip count is known only at run time,
so a thread has one block of weights in flight. With Q8_0's 34-byte blocks a V100 reads the 6144 -> 2560 projections of
Qwen3.8-Flash-Next (`ssm_out` x36 and `attn_output` x12, 48 matrices a verify window) at about 530-570 GB/s at 3
columns.
`STRATA_Q8_V70=1` makes the trip count a template parameter and loads all of a thread's weights before the first dot.
It is off by default and runs only on sm_70. The output is bitwise the default kernel's.
Measured on one Tesla V100 32 GB (setup under Validation), 6144 -> 2560, weights cold, through `native_mmvq`:
| columns | 2 | 3 | 4 |
|---|---|---|---|
| default (us) | 26.5-27.8 | 29.4-31.2 | 34.3-36.3 |
| `STRATA_Q8_V70=1` (us) | 20.3-20.5 | 21.1-21.3 | 24.0-24.3 |
(`ssm_out`'s copy count, 4 runs per arm over two sessions; `attn_output` gives 26.4-26.7 -> 20.2-20.3, 29.2-29.4 ->
21.0-21.2 and 34.1-34.5 -> 23.9-24.1.) At 1 and 5+ columns the default kernel runs.
An earlier end-to-end run (same kernel code, older base; details under Validation) gave the same text 0.8-1.0% faster
decode on top of the companion hyper-connection switch.
## Related discussion
- Companion PR: #6603 (sm_70 hyper-connection read, `STRATA_GR_V70`). Independent; either can merge first.
- #6529 (open): its sm_70 default table sets `STRATA_TSUM=1`, and this switch is not taken with `STRATA_TSUM=1` (a
different reduction kernel). Measured with #6529's head and its 21-switch table set by hand: Q8_0 6144 -> 2560 at 2/3/4
columns 23.8 / 28.2 / 30.7 us with the table, 20.4 / 21.2 / 24.1 us with the table plus `STRATA_TSUM=0` and this switch.
The output bytes were the same in all three setups (main default, #6529 table, this switch), so TSUM gave the same bits
here, but I only know that for these shapes and data. If #6529 lands, this switch helps only with `STRATA_TSUM=0`, unless
the maintainers know TSUM is bitwise in general and the guard can go. #6529's `STRATA_Q8_INPLACE` (opt-in, changes the
sum order) was not measured.
- #6287 (open) adds Q8_0 to the interleaved path (`native_mmvq_il`) with an sm_70 row. With `STRATA_SM70_TABLE=1` the
callers that use `native_mmvq_il` (GDN and QSA output projections, the shared expert) would take that kernel and never
reach this one. Measured with #6287 merged onto this branch: interleaved Q8_0 23.8 / 29.2 / 27.0 us against this
switch's 20.3 / 21.1 / 23.9 us at 2/3/4 columns, both bitwise. If both land, the interleaved Q8_0 row for sm_70 should
give way when `STRATA_Q8_V70=1`; I can do that in whichever PR goes second. The branches merge without conflicts.
Seen on the way, for #6287 (bench only, not run in the engine): in that merged build `native_mmvq_il` threw
`std::invalid_argument` ("native MMVQ requires n_in > 0 and divisible by its block element count") for the Q8_0
640 -> 2560 shape at 2 columns, which looks like its `validate_shape(n_in, ncols, QK)` with QK = 256.
- #3411 / #4767 (sm_70 defaults tables with `STRATA_B6_MMVQ_ROWS=1`): B6 has no Q8_0 kernel, so this switch is taken with or
without B6. A table that turns on `STRATA_TSUM=1` turns this switch off (see #6529).
- #6359: its kernels touch other types (Q4_K/Q5_K/Q6_K interleaved rows, Q5_0 expert rows); no Q8_0 overlap found.
- No scope issue opened: opt-in, single architecture, no default change.
## Approach
- `q8_v70_kernel<F, NCOLS, NIT>`: the default kernel's thread layout (4 warps, 2 rows a block, kbx = tid / 4 + 32 it, the
same two ints and scale), NIT = n_in / 1024 iterations unrolled, every iteration's weights loaded with
`SmallTraits<Q80Block, 8>::load` before the first `apply`. Each (column, row) sum adds the same values in the same order;
the cross-warp sum and the xor tree are the default's.
- Taken from `launch_multi_n` after `b6_launch`, for 2-4 columns, n_in a multiple of 1024 (2..8 x 1024), even n_out,
`multi_exact` on, not with `STRATA_TSUM=1`. Everything else keeps its current kernel.
- Off by default. Device code exists only for sm_70 (an empty kernel elsewhere, never launched); the host takes it only on
a compute capability 7.0 card (`per_dev[16]`, as the other per-device checks in `native_mmvq.cu`).
- Dispatch statistics: `native_mmvq` counts kDense var 0 before it dispatches, so these calls are counted like the kernel
they replace.
- Second commit: the register entry from `tools/settings_register.py` (only this entry) and `docs/SETTINGS.md` from
`--docs`. One row in `docs/NVIDIA_V100.md`.
## Validation
Environment: 1x Tesla V100 32 GB (board Tesla PG500-216), PCIe 3.0 x8 link at test time, driver 580.178.04, CUDA 12.9.86,
Ubuntu 24.04, Core i7-6800K host, build `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70
-DSTRATA_EXPERIMENTAL_SM60=ON`, Release. Revision: this branch head, and main c6f6427d9 for the comparisons.
**Passed:**
- Build of every target on this branch alone (sm_70): passed.
- Timing: a standalone program (not part of this PR; I can attach it) that calls `native_mmvq` at the model's dense shapes
and types (21 cases, 1-5 columns), weights rotating over copies larger than L2, 20 warm-up and 300 timed calls with
CUDA events, `STRATA_SM70_TABLE=1 STRATA_GR_NORM_FAST=1`. Order off, on, on, off, in each of two sessions: the table
above. The output bytes of all 105 case/width pairs (every type, not only Q8_0) are the same with the switch off, with
it on, and on main.
- `mmvq_multi_parity` (its Q8_0 case is 2048 -> 512, so 2-4 columns take this kernel) with `STRATA_Q8_V70=1`: "ok", 0 of
148480 outputs differ from single-column calls; the negative control still finds its differences. The kernel announced
itself, so the path was taken.
- `ctest -R "gr_parity|hc_persist|mmvq_multi_parity|mmvq_il_parity"` (7 tests) with `STRATA_Q8_V70=0` and `=1`: 7/7 passed
both times.
- Sanity check: I reversed the kernel's block order (`it` from NIT - 1 down) and rebuilt. `mmvq_multi_parity` then
FAILED at Q8_0 with 2, 3 and 4 columns (633 / 969 / 1231 differing outputs), and the timing program's output bytes
changed at 2-4 columns only. Change reverted and rebuilt.
- Default path unchanged: with the switch unset the output bytes equal main's (above). In the sm_70 SASS of
`native_mmvq.cu`, every kernel matches main's except some `native_mmvq_il_pf_kernel` instances, whose FFMA operands are
commuted; the same kind of difference, in 11 such instances, shows up in a build of the companion branch, whose
`native_mmvq.cu` is byte-identical to main's, so it is build-to-build noise. The 21 new kernels are the only additions.
- `python tools/settings_register.py --check`: clean.
- HIP compile, on a separate machine (ROCm 7.2, gfx1030, no GPU run): `native_mmvq.cu` and `fused_gr.cu` compile with
exactly main's warnings, and the `strata` target links. The new code is CUDA-only (`#if !defined(__HIPCC__)`).
- Earlier end-to-end run (2026-11-22, before the rebase; the 21 new kernels are SASS-identical to this branch): v0.2.18
(6e38f588a), same V100 model but the x16 card, Qwen3.8-Flash-Next IQ3_S GGUF (`ssm_out` / `attn_output` Q8_0), settings
`STRATA_SM70_TABLE=1 STRATA_ARGMAX_ROWS_SM70=1 STRATA_GR_NORM_FAST=1 STRATA_PF_WMMA=1 STRATA_ROUTE_TAIL_SKIP=0`, MTP
drafting, greedy. Greedy output is not repeatable run to run on this box, so I compared only runs whose output md5
matched. Adding this switch on top of `STRATA_GR_V70=1`: decode 9182 -> 9094 ms and 9488 -> 9412 ms on two prompts
(-1.0%, -0.8%), same text. A profiled pair (both switches on vs off): GDN output projection 1.26 -> 1.07 ms and QSA output
projection 0.44 -> 0.39 ms per window.
**Failed:**
- `python tools/test_settings_register.py`: 1 failure (41 register entries without a status). It fails the same way on
main c6f6427d9; none of the 41 is from this PR.
**Not run:**
- End-to-end A/B on this branch head: the test machine's other GPU serves a model, and a second model server does not
fit in its RAM. The end-to-end numbers above come from the older base.
- A greedy-output comparison of main vs this branch with the switch unset: not possible here (greedy output differs run
to run on this box even on one build). The output bytes and SASS above are what I could show.
- SYCL: no oneAPI toolchain here; the SYCL build has its own sources, which this PR does not touch (read from the CMake
files; unconfirmed static analysis).
- `STRATA_NO_MMVQ_ROWS2=1` (one row a block): this kernel replaces that kernel too; the two should give the same bits (each
row keeps its own sum), but I did not run that pairing.
- Shapes with n_in other than 6144 and 2048 (any multiple of 1024 up to 8192 is taken) and any card other than this V100.
## Limits
- One V100, one model and pack. The gain is per Q8_0 GEMV at 2-4 columns; at 3 columns it is about 8 us for each of the 48
matrices, about 0.4 ms a window (arithmetic from the table, not measured end to end).
- Not taken with `STRATA_TSUM=1`, so a table that enables TSUM (as #6529's does) disables it.
- The register tool classifies the switch as "NVIDIA and AMD" in `docs/SETTINGS.md`, because it sits in an
`#if !defined(__HIPCC__)` block; it never runs on AMD.
- This PR and the companion hyper-connection PR both change the switch count line in `docs/SETTINGS.md`; whichever merges
second needs `python tools/settings_register.py --docs` again.
- Written with AI assistance (Claude, through Claude Code): the code, the test scripts, the runs on my V100 and this text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。