#6359 [Benchmark] Tesla V100 (sm_70): opt-in decode kernels on 0.2.16, decode +13..24% on 2x V100-SXM2-32GB
open · @ksooad · 0 コメント · GitHub で見る
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindows
本文
## Summary
I run Strata as a coding-agent backend on two Tesla V100-SXM2-32GB. Over the last weeks I wrote a set of
**opt-in, sm_70-only decode kernels** on top of the release (carried from 0.2.12 → 0.2.14 → 0.2.16, each time the
commits applied cleanly). On 0.2.16 they give **+13..24% decode** over the stock engine with the same settings,
with no change to prompt speed.
Branch (9 commits on top of `v0.2.16`: one bug fix, then one commit per switch): https://github.com/ksooad/Strata/tree/v100-sm70-0216
I am not opening a PR. I'm posting the measurements and the branch so maintainers and other V100 owners can take
whatever is useful. I'll answer questions here. The work was AI-assisted (Claude), as the commit trailers say.
While doing this I also measured the **0.2.16 no-MMQ prompt path on a K-quant model (UD-Q3_K_XL)** on the V100,
which the release notes did not cover. Results are in the second table below.
## Related
- #6088 (V100 prompt −11% after the own-kernel move): the prompt table below adds K-quant numbers on 0.2.16, where
the Volta int8 fixes recovered short and mid prompts but long prompts are still ~9% behind MMQ.
- #3411 / #4767 (V100 defaults tables): my environment already uses those existing switches (`STRATA_B6_MMVQ_ROWS`,
`STRATA_SM70_TABLE`, ...). The switches here are new code on top of them, not a different table.
- #6286 / #6287 (Q8_0 in the interleaved MMVQ path): this touches the same `native_mmvq.cu` tables as my
`STRATA_MMVQ_IL_PF` commit, so the two would need a merge.
- The `STRATA_VERIFY_DEVICE_PLAN` PLE-wait bug I found on the way is reported separately: #6357.
## Setup
- 2x Tesla V100-SXM2-32GB on PCIe 3.0 x16 adapters (no NVLink, no P2P, `nvidia-smi topo` = SYS), power limit 225 W,
driver 580.178.04
- 2x Xeon E5-2680 v4, 8x16 GB DDR4-2400, Ubuntu 26.04
- CUDA 12.8 (12.8.93), host g++-14
- Model: Qwen3.8-Flash-Next **UD-Q3_K_XL** (Unsloth), layer split over both cards (`layer_split auto`), 256K context,
`--kv int8`, `--kv-resident 106496` (with it, 100% of the experts are resident on both cards), MTP `--spec 4 --spec-min-p 0.5`,
`--pipeline-windows 2`, `presence_penalty 0`, `--pool-workers 27`, `numactl --interleave=all`
- Build (identical for the stock and the patched engine):
```
cmake -S engine -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=70 \
-DSTRATA_ENABLE_CUDA=ON -DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_BUILD_TESTS=OFF -DSTRATA_Q6K_EXPERTS=ON \
-DCMAKE_CUDA_COMPILER=<cuda-12.8>/bin/nvcc -DCMAKE_CUDA_HOST_COMPILER=g++-14
cmake --build build --target strata
```
Method: my own script, not the COMMUNITY_BENCHMARKS harness. It uses the server's `timings`, greedy (temperature 0),
thinking off, and 512 generated tokens per prompt. There are 4 decode prompts (Russian explanation, English story,
Python, SQL), 2 reps each. The prompt runs use fresh filler with no prefix reuse. Every arm was loaded, warmed up and
measured in its own server process, and arms were interleaved over 2 rounds. Rounds agreed within 0.3%.
## Decode, 0.2.16 (tok/s, median of 2 reps, both rounds equal within 0.2)
| prompt | stock 0.2.16, default env | stock 0.2.16 + my env | **patched 0.2.16 + my env** | vs stock + env |
|---|---|---|---|---|
| ru-explain | 99.7 | 113.8 | **128.4** | +12.8% |
| en-story | 85.0 | 94.9 | **110.7** | +16.6% |
| code-py | 116.5 | 126.0 | **156.8** | +24.4% |
| code-sql | 129.8 | 150.5 | **173.9** | +15.5% |
"My env" is the same environment in both arms; the stock engine simply does not have the new switches. It also
contains existing upstream switches (`STRATA_SM70_TABLE=1 STRATA_FUSE_DECODE=1 STRATA_PF_WMMA=1 STRATA_HC_Q8=0 ...`,
full list below). Those explain the stock-default → stock+env column.
Prompt speed is unchanged by the patches: 4K 1341 → 1343, 30K 1455 → 1456, 100K 2046 → 2046 tok/s.
## Prompt, 0.2.14 + MMQ vs 0.2.16 (no MMQ), same machine and model (tok/s)
The 0.2.14 arm is my patched 0.2.14 built with `-DSTRATA_MMQ=ON`, and the 0.2.16 arm is my patched 0.2.16.
Decode is equal in the two (156.9 vs 156.5 median), and the patches do not touch the prompt path.
| new tokens | 0.2.14 + MMQ | 0.2.16 int8 kernels | |
|---|---|---|---|
| 600 | 373 | 711 | +90% |
| 1K | 539 | 934 | +73% |
| 2K | 809 | 1146 | +42% |
| 4K | 1111 | 1343 | +21% |
| 8K | 1377 | 1441 | +5% |
| 16K | 1495 | 1437 | −4% |
| 30K | 1597 | 1456 | −9% |
| 100K | 2255 | 2046 | −9% |
So on this K-quant model the int8 path is a big win below ~12K new tokens and about 9% behind MMQ on long prompts.
For an agent workload that's a clear net gain. In my server log, 52% of the prompt time goes to reads under 4K new
tokens. I also measured 0.2.14 built without MMQ: it was 16% slower than with MMQ at 30K, so #6053 and the Volta tile
rule recovered most of that gap.
## What the switches do (all off by default, all sm_70 code paths)
| switch | what | exactness | decode gain measured on 0.2.14 (geo-mean) |
|---|---|---|---|
| `STRATA_GR_V70=1` | hyper-connection read for sm_70: two kernels (down + latency-hidden up), conflict-free planes, L2 prefetch; −1.24 ms per decode window | **bounded, not bitwise** (sums in another order), see below | 6.6% |
| `STRATA_MMVQ_IL_PF=1` | interleaved GEMV: W-warp row-group kernel + an sm_70 pick table (Q4_K/Q5_K/Q6_K shapes, head) | bitwise (`mmvq_il_parity`, T 1-8) | 1.4% |
| `STRATA_DOWN_LDS_Q50=1` | Q5_0 expert down rows through the LDS down kernel | bitwise | 1.5% |
| `STRATA_IL_FAST` (compile-time, default 1) + `STRATA_GU8_MINB` (compile-time, default 3) | exact integer forms in the base IL `apply()` (no spill on sm_70); gate/up mode-8 kernel compiled for 3 blocks/SM | bitwise | 1.9% (both) |
| `STRATA_EXP_PF=6` | L2 prefetch of the next pass rows in the mode-8 expert kernels; bits 6-9 = gate/up with 1-2 entries in the k loop | bitwise | 1.06% |
| `STRATA_ROUTE_PLAN_FUSE=1` | router top-k and the resident plan in one launch | bitwise (`route_plan_parity`) | 0.66% |
| `STRATA_XQ_SIDE=1` | experts' q8_1 image on the shared expert stream | bitwise | noise |
| `STRATA_VERIFY_DEVICE_PLAN=2` | plan + doorbell in one launch for partly resident layers | bitwise | +9.2% together with `DOWN_LDS_Q50` on 0.2.12 with ~91% of the experts resident; ~0 at 100% residency, so **off** in the runs above |
On 0.2.14 the whole set took decode from 108.5 to 126.2 tok/s (geo-mean, +16%). On 0.2.16 it gives +13..24% per
prompt (table above).
Note: `STRATA_HC_Q8=0` is needed with UD files on 0.2.14+, because the engine turns HC_Q8 on by itself, and that
path bypasses `GR_V70` (−3..4% decode).
## Exactness checks (0.2.16, 6 greedy prompts × 400 tokens: ru, en, py, sql, go, math)
- Patched engine with **no env** vs stock 0.2.16 with no env: **6 of 6 texts identical**.
- Patched engine with my env vs the same patches on 0.2.14 + MMQ: 6 of 6 identical.
- Patched + env vs stock: texts diverge (first difference between character 4 and 1388). This is expected from
`STRATA_GR_V70`, which is bounded rather than bitwise. `hc_v70_parity` (in the branch) checks it against the default read and an FP64
host reference at the real geometry (n_embd 2560, hc 4, T = 1..8). Every output is finite and within 1e-4 of the
default read, relative to its largest magnitude. Its FP64 error is no worse than 2× the default read's, two runs
are bitwise equal, and its q8_1 bytes are within one step. PASS for all 8 token counts.
- No separate quality eval was run on the patched engine.
## Limits
- `STRATA_IL_FAST` and `STRATA_GU8_MINB` are compile-time defaults and are **not gated on `__CUDA_ARCH__`**, so they
change the generated code on every CUDA arch. They are bitwise by construction, but I have only measured
performance on the V100. They should be gated to sm_70 (or flipped to 0) before anyone takes them.
- One machine, one model (UD-Q3_K_XL), two-card layer split, all experts resident. The −1.24 ms/window figure for
GR_V70 is from an earlier release on one card with IQ3_S; DEVICE_PLAN=2 +9.2% is from 0.2.12 with UD-Q3_K_XL.
- Experimental code that is off and not faster is still in the branch (`STRATA_GR_V70_UPPF`, the fused one-kernel modes
2/3, `STRATA_IL_SMEM`). It can be dropped.
<details><summary>Full STRATA_* environment of the measured runs</summary>
```
STRATA_GR_V70=1 STRATA_MMVQ_IL_PF=1 STRATA_DOWN_LDS_Q50=1 STRATA_EXP_PF=6 STRATA_ROUTE_PLAN_FUSE=1 STRATA_XQ_SIDE=1
STRATA_VERIFY_DEVICE_PLAN=0 STRATA_HC_Q8=0 STRATA_SMALL_UNR=3 STRATA_SPLIT_OVERLAP=1
STRATA_SM70_TABLE=1 STRATA_PF_WMMA=1 STRATA_FUSE_DECODE=1 STRATA_ARGMAX_ROWS_SM70=1 STRATA_GR_NORM_FAST=1
STRATA_B6_MMVQ_ROWS=1 STRATA_QIL=1 STRATA_GU_SWIGLU=1 STRATA_EXPDEC=1 STRATA_QSA_ROWS=1 STRATA_SIDE_MIXER=2
STRATA_FUSE_WAIT=2 STRATA_VERIFY_QDEDUP=1 STRATA_ATTN_DAILY=1 STRATA_CPU_IQ_ALL_EXACT1=1
STRATA_ADAPT_FETCH=2 STRATA_ADAPT_LAG=2 STRATA_ADAPT_NOWAIT=1 STRATA_ROUTE_TAIL_SKIP=0 STRATA_ROUTE_PRIOR=0
STRATA_GDN_TC=0 STRATA_GDN_CONVL2=0 STRATA_NUMA_POOL=0
```
</details>
関連リンク
インストール・モデル・リリースへのサイト内リンク。