#6603 [Engine] sm_70: token-split hyper-connection read (STRATA_GR_V70, opt-in, bitwise)

open · @danexpo · 0 comments · View on GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentation

Description

## Problem and result

On a V100 the hyper-connection read keeps one warp per row for all tokens of a verify window. The down projection runs
on 41 blocks of 8 rows, so half of the 80 SMs idle. sm_70 has no cp.async, so the staged read cannot hide the
activation loads. Each extra token adds latency-bound work.

`STRATA_GR_V70=1` gives each (row, token) pair its own warp. It is off by default and runs only on sm_70. The output is
bitwise the default read's.

Before and after, measured on one Tesla V100 32 GB (see Validation for the setup), one read with 48 cold weight sets:

| tokens | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| default read (us) | 37.0 | 43.6-43.8 | 48.7-48.8 | 55.8-55.9 | 64.1-64.2 | 68.3-68.4 | 79.0 | 87.3 |
| `STRATA_GR_V70=1` (us) | 30.3 | 39.6-39.7 | 44.0-44.1 | 49.7-49.8 | 56.6 | 62.9 | 74.5-74.6 | 81.6 |
| change | -18% | -9% | -10% | -11% | -12% | -8% | -6% | -7% |

An earlier end-to-end run (same kernel code, older base; details under Validation) gave the same text about 2.6% faster
decode.

## Related discussion

- Companion PR: #6604 (sm_70 Q8_0 GEMV, `STRATA_Q8_V70`). Independent; either can merge first.
- #6359 (benchmark report, branch only, no PR) has a different sm_70 hyper-connection read under the same name,
  `STRATA_GR_V70`. That one is "bounded, not bitwise" and its author measured a larger gain (-1.24 ms per window). This PR
  is bitwise. If both are wanted, one of the two names has to change; I can rename this one.
- #6529 (open) adds Q8_0 hyper-connection kernels (`STRATA_GR_Q8_*_MODE`). They run only with Q8_0 hyper-connection
  weights (`STRATA_HC_Q8=1`, opt-in on CUDA and not in #6529's sm_70 table). This read runs only with BF16 weights. The
  two never run in the same configuration. Measured with #6529's head and its 21-switch sm_70 table set by hand, this
  read keeps its gain (36.9 -> 30.2 us at 1 token, 87.0 -> 81.6 us at 8) with the same output hashes. #6529's Q8_0 read is
  faster from 2 tokens on (for example 42.1 us at 4 tokens against 49.8 us here), but it changes the weights to 8 bits, so its
  output is not the BF16 read's.
- #3411 / #4767 (sm_70 defaults tables): no interaction found. This switch stays opt-in and is not proposed for a table.
- No scope issue opened: this is an opt-in, single-architecture kernel with no default change.

## Approach

- Down projection: one block per row (320 rows, plus 4 inject rows), one warp per token. Lane L sums chunks L + 32q in
  ascending q with the same 8-FMA dot and the same xor butterfly as the plain and staged down kernels.
- Up projection: N/2 blocks of 2 columns x 4 streams x T tokens, 8 threads per (row, token). The group of 8 does
  `gr_up_fast_kernel`'s lanes j, j+8, j+16, j+24 and the tail chunk 32 + j with the same xor tree. The epilogues are the
  default ones. The norm is the chosen variant's own.
- Only who computes what changes: more warps in flight, and the T warps of a row read its weights through L1.
- Off by default. Device code exists only for sm_70 (other architectures get empty kernels that are never launched); the
  host takes the path only on a compute capability 7.0 card.
- Compiled for the shape it was written for (hc_lr 320, hc 4: flash_next and q4_deep). The other geometries build and keep
  the default read.
- Not taken with Q8_0 hyper-connection weights (`STRATA_HC_Q8`, `STRATA_HC_REQ8`) or the q8_1 tail (`STRATA_QFUSE`).
- `fused_gr_check` now also runs this read under the plain, split and staged variants and compares each with that
  variant's default read, bit for bit, at 1-8 tokens with and without the pending write. On any difference it prints the
  first one and keeps the default read on that card. The check also runs when `STRATA_HC_SPLIT=0`. With the switch unset,
  the check does exactly what it did before.
- Counted in the dispatch statistics as kHc var 30 (the stream-split read uses 20 + split).
- Second commit: the register entry from `tools/settings_register.py` (only this entry; a full regeneration on main also
  rewrites about 1,400 unrelated lines) and `docs/SETTINGS.md` from `--docs`. One row in `docs/NVIDIA_V100.md`.

## Validation

Environment: 1x Tesla V100 32 GB (board Tesla PG500-216), PCIe 3.0 x8 link at test time, driver 580.178.04, CUDA 12.9.86,
Ubuntu 24.04, Core i7-6800K host, build `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 -DSTRATA_EXPERIMENTAL_SM60=ON`, Release.
Kernel runs used `STRATA_SM70_TABLE=1 STRATA_ARGMAX_ROWS_SM70=1 STRATA_GR_NORM_FAST=1 STRATA_PF_WMMA=1
STRATA_ROUTE_TAIL_SKIP=0`. Revision: this branch head, and main c6f6427d9 for the comparisons.

**Passed:**

- Build of every target on this branch alone (sm_70): passed.
- `fused_gr_bench 200 1 8 48` with `STRATA_HC_SPLIT=2` (the staged read the engine picks on this card after its check;
  without it the bench reads with the plain variant), four runs in the order off, on, on, off, in each of two sessions:
  the times in the table above. All 32 output hashes (T 1-8 x pending write x inject) are the same with the switch off,
  with it on, and on main.
- `hc_persist_parity --off` with `STRATA_GR_V70=1` (it calls `fused_gr_check`), with `STRATA_HC_SPLIT` unset, 0 and 1:
  "checked bit for bit against the default read at 1-8 tokens" each time.
- `ctest -R "gr_parity|hc_persist|mmvq_multi_parity|mmvq_il_parity"` (7 tests) with `STRATA_GR_V70=0` and `=1`: 7/7 passed
  both times.
- Sanity check of the new check: I changed the up kernel's sum order to `(p0 + p1) + (p2 + p3)` and rebuilt.
  `fused_gr_check` then reported "STRATA_GR_V70=1 is off here: the token-split read differs from the default read: under
  the plain read: mixed differs for 1 token(s) at 2 (bdc0deae, not bdc0deb0)", `gr_parity` reported 2 failures and the
  bench hashes changed. Change reverted and rebuilt.
- Default path unchanged: with the switch unset the bench hashes equal main's. The sm_70 SASS of every kernel that main's
  `fused_gr.cu` has is identical in this branch (cuobjdump, names normalised); the only new kernels are the 16 of this read.
- `tools/check_cuda_spills.py --list`: the new kernels have STACK 0 and LOCAL 0.
- `python tools/settings_register.py --check`: clean.
- HIP compile, on a separate machine (ROCm 7.2, gfx1030, no GPU run): `fused_gr.cu` and `native_mmvq.cu` compile with
  exactly main's warnings, and the `strata` target links. The new code is CUDA-only (`#if !defined(__HIPCC__)`).
- Earlier end-to-end run (2026-11-22, before the rebase; the 16 new kernels are SASS-identical to this branch): v0.2.18
  (6e38f588a) plus this read, same V100 model but the x16 card, Qwen3.8-Flash-Next IQ3_S GGUF (BF16 hyper-connection
  tensors), the settings above, MTP drafting, greedy, 3 prompts. Greedy output is not repeatable run to run on this box,
  so I compared only runs whose output md5 matched. The code prompt: decode 9709 / 9687 / 9809 ms default vs 9457 ms with
  `STRATA_GR_V70=1` (-2.6% against their mean), same text. Arm means over 3 prompts: 72.2 vs 73.3 tok/s (+1.5%; the
  ranges overlap, 2 arms each). A profiled pair (this read and the companion Q8_0 switch on; the Q8_0 one does not touch
  these stages): the hyper-connection stages of a window 5.58 -> 4.98 ms.

**Failed:**

- `python tools/test_settings_register.py`: 1 failure (41 register entries without a status). It fails the same way on
  main c6f6427d9; none of the 41 is from this PR.

**Not run:**

- End-to-end A/B on this branch head: the test machine's other GPU serves a model, and a second model server does not
  fit in its RAM. The end-to-end numbers above come from the older base.
- A greedy-output comparison of main vs this branch with the switch unset: not possible here (greedy output differs run
  to run on this box even on one build). The kernel hashes and SASS above are what I could show.
- SYCL: no oneAPI toolchain here. The SYCL build uses its own `fused_gr.dp.cpp`, which this PR does not touch (read from
  the CMake files; unconfirmed static analysis).
- Other geometries (q4_lite, q4_mid, q4_big, q4_deep): the code is written so they compile and keep the default read, but
  I did not build them.
- Multi-GPU layer splits and any card other than this one V100.

## Limits

- One V100, one model and pack. The kernel gain shrinks with the token count (-18% at 1 token, -6% to -7% at 7-8).
- With Q8_0 hyper-connection weights (`STRATA_HC_Q8=1`, or a pack the engine reads that way) this read is not taken.
- The register tool classifies the read as "NVIDIA and AMD" in `docs/SETTINGS.md`, because it sits in an
  `#if !defined(__HIPCC__)` block; it never runs on AMD.
- This PR and the companion Q8_0 PR both change the switch count line in `docs/SETTINGS.md`; whichever merges second
  needs `python tools/settings_register.py --docs` again.
- Written with AI assistance (Claude, through Claude Code): the code, the test scripts, the runs on my V100 and this text.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.