#6603 [Engine] sm_70: token-split hyper-connection read (STRATA_GR_V70, opt-in, bitwise)
open · @danexpo · 0 commentaires · Sur GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentation
Description
## Problem and result On a V100 the hyper-connection read keeps one warp per row for all tokens of a verify window. The down projection runs on 41 blocks of 8 rows, so half of the 80 SMs idle. sm_70 has no cp.async, so the staged read cannot hide the activation loads. Each extra token adds latency-bound work. `STRATA_GR_V70=1` gives each (row, token) pair its own warp. It is off by default and runs only on sm_70. The output is bitwise the default read's. Before and after, measured on one Tesla V100 32 GB (see Validation for the setup), one read with 48 cold weight sets: | tokens | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |---|---|---|---|---|---|---|---|---| | default read (us) | 37.0 | 43.6-43.8 | 48.7-48.8 | 55.8-55.9 | 64.1-64.2 | 68.3-68.4 | 79.0 | 87.3 | | `STRATA_GR_V70=1` (us) | 30.3 | 39.6-39.7 | 44.0-44.1 | 49.7-49.8 | 56.6 | 62.9 | 74.5-74.6 | 81.6 | | change | -18% | -9% | -10% | -11% | -12% | -8% | -6% | -7% | An earlier end-to-end run (same kernel code, older base; details under Validation) gave the same text about 2.6% faster decode. ## Related discussion - Companion PR: #6604 (sm_70 Q8_0 GEMV, `STRATA_Q8_V70`). Independent; either can merge first. - #6359 (benchmark report, branch only, no PR) has a different sm_70 hyper-connection read under the same name, `STRATA_GR_V70`. That one is "bounded, not bitwise" and its author measured a larger gain (-1.24 ms per window). This PR is bitwise. If both are wanted, one of the two names has to change; I can rename this one. - #6529 (open) adds Q8_0 hyper-connection kernels (`STRATA_GR_Q8_*_MODE`). They run only with Q8_0 hyper-connection weights (`STRATA_HC_Q8=1`, opt-in on CUDA and not in #6529's sm_70 table). This read runs only with BF16 weights. The two never run in the same configuration. Measured with #6529's head and its 21-switch sm_70 table set by hand, this read keeps its gain (36.9 -> 30.2 us at 1 token, 87.0 -> 81.6 us at 8) with the same output hashes. #6529's Q8_0 read is faster from 2 tokens on (for example 42.1 us at 4 tokens against 49.8 us here), but it changes the weights to 8 bits, so its output is not the BF16 read's. - #3411 / #4767 (sm_70 defaults tables): no interaction found. This switch stays opt-in and is not proposed for a table. - No scope issue opened: this is an opt-in, single-architecture kernel with no default change. ## Approach - Down projection: one block per row (320 rows, plus 4 inject rows), one warp per token. Lane L sums chunks L + 32q in ascending q with the same 8-FMA dot and the same xor butterfly as the plain and staged down kernels. - Up projection: N/2 blocks of 2 columns x 4 streams x T tokens, 8 threads per (row, token). The group of 8 does `gr_up_fast_kernel`'s lanes j, j+8, j+16, j+24 and the tail chunk 32 + j with the same xor tree. The epilogues are the default ones. The norm is the chosen variant's own. - Only who computes what changes: more warps in flight, and the T warps of a row read its weights through L1. - Off by default. Device code exists only for sm_70 (other architectures get empty kernels that are never launched); the host takes the path only on a compute capability 7.0 card. - Compiled for the shape it was written for (hc_lr 320, hc 4: flash_next and q4_deep). The other geometries build and keep the default read. - Not taken with Q8_0 hyper-connection weights (`STRATA_HC_Q8`, `STRATA_HC_REQ8`) or the q8_1 tail (`STRATA_QFUSE`). - `fused_gr_check` now also runs this read under the plain, split and staged variants and compares each with that variant's default read, bit for bit, at 1-8 tokens with and without the pending write. On any difference it prints the first one and keeps the default read on that card. The check also runs when `STRATA_HC_SPLIT=0`. With the switch unset, the check does exactly what it did before. - Counted in the dispatch statistics as kHc var 30 (the stream-split read uses 20 + split). - Second commit: the register entry from `tools/settings_register.py` (only this entry; a full regeneration on main also rewrites about 1,400 unrelated lines) and `docs/SETTINGS.md` from `--docs`. One row in `docs/NVIDIA_V100.md`. ## Validation Environment: 1x Tesla V100 32 GB (board Tesla PG500-216), PCIe 3.0 x8 link at test time, driver 580.178.04, CUDA 12.9.86, Ubuntu 24.04, Core i7-6800K host, build `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON`, Release. Kernel runs used `STRATA_SM70_TABLE=1 STRATA_ARGMAX_ROWS_SM70=1 STRATA_GR_NORM_FAST=1 STRATA_PF_WMMA=1 STRATA_ROUTE_TAIL_SKIP=0`. Revision: this branch head, and main c6f6427d9 for the comparisons. **Passed:** - Build of every target on this branch alone (sm_70): passed. - `fused_gr_bench 200 1 8 48` with `STRATA_HC_SPLIT=2` (the staged read the engine picks on this card after its check; without it the bench reads with the plain variant), four runs in the order off, on, on, off, in each of two sessions: the times in the table above. All 32 output hashes (T 1-8 x pending write x inject) are the same with the switch off, with it on, and on main. - `hc_persist_parity --off` with `STRATA_GR_V70=1` (it calls `fused_gr_check`), with `STRATA_HC_SPLIT` unset, 0 and 1: "checked bit for bit against the default read at 1-8 tokens" each time. - `ctest -R "gr_parity|hc_persist|mmvq_multi_parity|mmvq_il_parity"` (7 tests) with `STRATA_GR_V70=0` and `=1`: 7/7 passed both times. - Sanity check of the new check: I changed the up kernel's sum order to `(p0 + p1) + (p2 + p3)` and rebuilt. `fused_gr_check` then reported "STRATA_GR_V70=1 is off here: the token-split read differs from the default read: under the plain read: mixed differs for 1 token(s) at 2 (bdc0deae, not bdc0deb0)", `gr_parity` reported 2 failures and the bench hashes changed. Change reverted and rebuilt. - Default path unchanged: with the switch unset the bench hashes equal main's. The sm_70 SASS of every kernel that main's `fused_gr.cu` has is identical in this branch (cuobjdump, names normalised); the only new kernels are the 16 of this read. - `tools/check_cuda_spills.py --list`: the new kernels have STACK 0 and LOCAL 0. - `python tools/settings_register.py --check`: clean. - HIP compile, on a separate machine (ROCm 7.2, gfx1030, no GPU run): `fused_gr.cu` and `native_mmvq.cu` compile with exactly main's warnings, and the `strata` target links. The new code is CUDA-only (`#if !defined(__HIPCC__)`). - Earlier end-to-end run (2026-11-22, before the rebase; the 16 new kernels are SASS-identical to this branch): v0.2.18 (6e38f588a) plus this read, same V100 model but the x16 card, Qwen3.8-Flash-Next IQ3_S GGUF (BF16 hyper-connection tensors), the settings above, MTP drafting, greedy, 3 prompts. Greedy output is not repeatable run to run on this box, so I compared only runs whose output md5 matched. The code prompt: decode 9709 / 9687 / 9809 ms default vs 9457 ms with `STRATA_GR_V70=1` (-2.6% against their mean), same text. Arm means over 3 prompts: 72.2 vs 73.3 tok/s (+1.5%; the ranges overlap, 2 arms each). A profiled pair (this read and the companion Q8_0 switch on; the Q8_0 one does not touch these stages): the hyper-connection stages of a window 5.58 -> 4.98 ms. **Failed:** - `python tools/test_settings_register.py`: 1 failure (41 register entries without a status). It fails the same way on main c6f6427d9; none of the 41 is from this PR. **Not run:** - End-to-end A/B on this branch head: the test machine's other GPU serves a model, and a second model server does not fit in its RAM. The end-to-end numbers above come from the older base. - A greedy-output comparison of main vs this branch with the switch unset: not possible here (greedy output differs run to run on this box even on one build). The kernel hashes and SASS above are what I could show. - SYCL: no oneAPI toolchain here. The SYCL build uses its own `fused_gr.dp.cpp`, which this PR does not touch (read from the CMake files; unconfirmed static analysis). - Other geometries (q4_lite, q4_mid, q4_big, q4_deep): the code is written so they compile and keep the default read, but I did not build them. - Multi-GPU layer splits and any card other than this one V100. ## Limits - One V100, one model and pack. The kernel gain shrinks with the token count (-18% at 1 token, -6% to -7% at 7-8). - With Q8_0 hyper-connection weights (`STRATA_HC_Q8=1`, or a pack the engine reads that way) this read is not taken. - The register tool classifies the read as "NVIDIA and AMD" in `docs/SETTINGS.md`, because it sits in an `#if !defined(__HIPCC__)` block; it never runs on AMD. - This PR and the companion Q8_0 PR both change the switch count line in `docs/SETTINGS.md`; whichever merges second needs `python tools/settings_register.py --docs` again. - Written with AI assistance (Claude, through Claude Code): the code, the test scripts, the runs on my V100 and this text. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.