Issues / #2782
#2782 sm_120 (RTX PRO 6000 Blackwell): mmvq_il_parity --emit-table over 7 runs, decode +1.3% in the engine; the table would also apply to the RTX 5070
open · @brocktice · 0 コメント · GitHub で見る
BenchmarksNVIDIA / CUDAModels & quantsDocumentationLinux
本文
## Card
RTX PRO 6000 Blackwell Max-Q Workstation Edition (sm_120, 96 GB, 300 W), driver 595.91.07, CUDA 13.2, Linux. Strata v0.1.43 built with `-DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_BUILD_TESTS=ON`.
## mmvq_il_parity --bench --emit-table
Seven runs on an otherwise idle card; every run ends `mmvq_il_parity: OK (every case, T 1-4, rows 1/2/4 and the table, bitwise)`. Several cells flip between runs, so the block below applies `emit_table`'s rule (worst shape of a cell under 0.97 of native_mmvq, else 0; a class with no shape copies its lower neighbour) to the **median** of the seven runs per shape, token count and rows choice:
```cpp
{23, {{0, 1, 1, 2, 1}, {0, 4, 2, 4, 4}, {0, 2, 2, 2, 2}}}, // IQ4_XS
{12, {{0, 0, 1, 2, 2}, {0, 2, 2, 4, 4}, {0, 2, 2, 2, 2}}}, // Q4_K
{13, {{0, 0, 1, 4, 4}, {0, 1, 2, 4, 2}, {0, 2, 2, 2, 2}}}, // Q5_K
{14, {{0, 0, 1, 4, 4}, {0, 0, 2, 4, 1}, {0, 2, 2, 2, 2}}}, // Q6_K
{11, {{0, 1, 2, 2, 4}, {0, 2, 2, 4, 2}, {0, 2, 2, 2, 2}}}, // Q3_K
// one column:
{23, {0, 0, 1, 0, 2}}, // IQ4_XS
{12, {0, 0, 1, 1, 1}}, // Q4_K
{13, {0, 0, 1, 1, 1}}, // Q5_K
{14, {0, 0, 0, 0, 0}}, // Q6_K
{11, {0, 0, 2, 2, 2}}, // Q3_K
```
Summed over every measured shape at 2-4 columns, these rows take 13.9% less time than `kIlRowsShared`'s picks on this card (median times): IQ4_XS -7.0%, Q4_K -7.3%, Q5_K -3.8%, Q6_K -8.4%, and Q3_K -34%, which the shared table leaves on native_mmvq (-38 to -49% at 6144-12288 rows, -12 to -20% at 2560, none below 2048).
The full output of run 1 is below; I can attach all seven.
## Decode A/B in the engine
Qwen3.8-Flash-Next IQ3_S, all experts resident, `--batch 4 --batch-mtp --spec 4 --kv int8`, one card, greedy, thinking off. Each arm is a fresh server; four fixed prompts (256 tokens, the median of the 2nd and 3rd pass), one ~8K-token prompt, and the four prompts at once. Arm B sets `STRATA_MMVQ_IL_ROWS` to the 2-4 column rows above (not the one-column rows); A is unset.
| pair (A/B interleaved) | solo decode B/A | 4 at once B/A | long prompt B/A |
|---|---|---|---|
| 1 | 1.013 | 1.026 | 0.997 |
| 2 | 1.011 | 0.999 | 1.002 |
| 3 | 1.015 | 0.987 | 0.999 |
A ran at 226-229 tok/s solo. All answers are byte-identical between the arms (and between every A run). With the one-column rows as well, a single screening run gave no extra gain. That is 3 pairs, not the 10 the doc asks for; I can run more if useful.
## The catch: sm_120 is also the RTX 5070
`kIlArch` is keyed by compute capability, so a `kIlRows_120` from this card would also replace the RTX 5070's half of `kIlRowsShared`. I have no 5070 to check it on, which is why this is an issue and not a PR. If a 5070 run of `--bench` exists, applying the same worst-of-both-cards rule to the two would give a table for both; the Q3_K row alone would not conflict with anything, since the shared table has none.
<details><summary>run 1, full output</summary>
```
BENCH IQ4_XS 2560x10240 T=1 multi 11.0 us | il rows1 11.3 rows2 10.8 rows4 11.5 us
BENCH IQ4_XS 2560x10240 T=2 multi 13.0 us | il rows1 11.8 rows2 11.3 rows4 11.6 us
BENCH IQ4_XS 2560x10240 T=3 multi 15.7 us | il rows1 14.6 rows2 13.4 rows4 12.8 us
BENCH IQ4_XS 2560x10240 T=4 multi 18.8 us | il rows1 13.6 rows2 12.4 rows4 12.8 us
BENCH IQ4_XS 2560x6144 T=1 multi 7.7 us | il rows1 7.1 rows2 7.2 rows4 9.0 us
BENCH IQ4_XS 2560x6144 T=2 multi 8.7 us | il rows1 7.4 rows2 7.5 rows4 8.0 us
BENCH IQ4_XS 2560x6144 T=3 multi 10.2 us | il rows1 9.2 rows2 8.3 rows4 8.2 us
BENCH IQ4_XS 2560x6144 T=4 multi 12.3 us | il rows1 8.7 rows2 8.2 rows4 8.4 us
BENCH IQ4_XS 6144x2560 T=1 multi 7.1 us | il rows1 7.1 rows2 8.3 rows4 14.0 us
BENCH IQ4_XS 6144x2560 T=2 multi 8.0 us | il rows1 7.4 rows2 8.3 rows4 8.1 us
BENCH IQ4_XS 6144x2560 T=3 multi 10.1 us | il rows1 12.3 rows2 11.3 rows4 8.8 us
BENCH IQ4_XS 6144x2560 T=4 multi 11.1 us | il rows1 9.9 rows2 8.6 rows4 8.7 us
BENCH IQ4_XS 2560x512 T=1 multi 3.0 us | il rows1 3.2 rows2 3.2 rows4 5.1 us
BENCH IQ4_XS 2560x512 T=2 multi 3.4 us | il rows1 3.4 rows2 3.6 rows4 3.5 us
BENCH IQ4_XS 2560x512 T=3 multi 3.5 us | il rows1 5.9 rows2 5.0 rows4 3.5 us
BENCH IQ4_XS 2560x512 T=4 multi 3.8 us | il rows1 4.7 rows2 3.6 rows4 3.7 us
BENCH Q4_K 2560x10240 T=1 multi 11.5 us | il rows1 11.0 rows2 11.0 rows4 12.0 us
BENCH Q4_K 2560x10240 T=2 multi 14.8 us | il rows1 15.2 rows2 12.3 rows4 17.2 us
BENCH Q4_K 2560x10240 T=3 multi 18.0 us | il rows1 14.1 rows2 13.3 rows4 12.2 us
BENCH Q4_K 2560x10240 T=4 multi 21.4 us | il rows1 14.3 rows2 13.8 rows4 16.5 us
BENCH Q4_K 6144x2560 T=1 multi 8.0 us | il rows1 8.4 rows2 9.5 rows4 21.0 us
BENCH Q4_K 6144x2560 T=2 multi 9.0 us | il rows1 11.6 rows2 14.8 rows4 32.5 us
BENCH Q4_K 6144x2560 T=3 multi 11.3 us | il rows1 11.4 rows2 10.0 rows4 10.7 us
BENCH Q4_K 6144x2560 T=4 multi 12.8 us | il rows1 10.9 rows2 10.6 rows4 29.0 us
BENCH Q5_K 2560x12288 T=1 multi 15.8 us | il rows1 15.3 rows2 15.3 rows4 15.8 us
BENCH Q5_K 2560x12288 T=2 multi 19.9 us | il rows1 17.0 rows2 18.8 rows4 16.2 us
BENCH Q5_K 2560x12288 T=3 multi 23.2 us | il rows1 18.0 rows2 17.8 rows4 19.2 us
BENCH Q5_K 2560x12288 T=4 multi 29.5 us | il rows1 19.8 rows2 19.1 rows4 30.0 us
BENCH Q5_K 2560x6144 T=1 multi 8.9 us | il rows1 8.5 rows2 8.7 rows4 11.1 us
BENCH Q5_K 2560x6144 T=2 multi 11.3 us | il rows1 9.5 rows2 9.5 rows4 10.0 us
BENCH Q5_K 2560x6144 T=3 multi 12.6 us | il rows1 9.7 rows2 9.6 rows4 9.8 us
BENCH Q5_K 2560x6144 T=4 multi 14.9 us | il rows1 11.4 rows2 9.8 rows4 15.6 us
BENCH Q5_K 6144x2560 T=1 multi 9.0 us | il rows1 9.0 rows2 13.3 rows4 21.9 us
BENCH Q5_K 6144x2560 T=2 multi 9.6 us | il rows1 10.5 rows2 12.6 rows4 15.7 us
BENCH Q5_K 6144x2560 T=3 multi 12.2 us | il rows1 11.1 rows2 12.1 rows4 11.7 us
BENCH Q5_K 6144x2560 T=4 multi 13.6 us | il rows1 11.7 rows2 11.4 rows4 31.7 us
BENCH Q5_K head 2560x248320 T=1 multi 335.6 us | il rows1 288.0 rows2 307.8 rows4 307.4 us
BENCH Q5_K head 2560x248320 T=2 multi 408.3 us | il rows1 358.5 rows2 333.5 rows4 327.6 us
BENCH Q5_K head 2560x248320 T=3 multi 483.4 us | il rows1 411.9 rows2 368.6 rows4 344.2 us
BENCH Q5_K head 2560x248320 T=4 multi 567.5 us | il rows1 431.8 rows2 386.6 rows4 411.8 us
BENCH Q6_K 2560x12288 T=1 multi 19.4 us | il rows1 20.0 rows2 30.1 rows4 29.1 us
BENCH Q6_K 2560x12288 T=2 multi 23.3 us | il rows1 21.8 rows2 27.8 rows4 19.9 us
BENCH Q6_K 2560x12288 T=3 multi 28.6 us | il rows1 21.7 rows2 22.0 rows4 23.6 us
BENCH Q6_K 2560x12288 T=4 multi 34.4 us | il rows1 27.7 rows2 23.5 rows4 26.5 us
BENCH Q6_K 2560x10240 T=1 multi 16.5 us | il rows1 18.2 rows2 19.3 rows4 28.2 us
BENCH Q6_K 2560x10240 T=2 multi 19.7 us | il rows1 20.1 rows2 17.7 rows4 17.6 us
BENCH Q6_K 2560x10240 T=3 multi 23.0 us | il rows1 19.6 rows2 19.8 rows4 17.3 us
BENCH Q6_K 2560x10240 T=4 multi 28.5 us | il rows1 21.5 rows2 20.0 rows4 23.8 us
BENCH Q6_K 6144x2560 T=1 multi 11.2 us | il rows1 17.2 rows2 32.0 rows4 58.4 us
BENCH Q6_K 6144x2560 T=2 multi 11.6 us | il rows1 18.4 rows2 28.1 rows4 26.8 us
BENCH Q6_K 6144x2560 T=3 multi 15.1 us | il rows1 18.5 rows2 15.1 rows4 16.5 us
BENCH Q6_K 6144x2560 T=4 multi 16.3 us | il rows1 19.1 rows2 14.9 rows4 21.1 us
BENCH Q6_K 2560x512 T=1 multi 3.3 us | il rows1 5.0 rows2 7.6 rows4 14.0 us
BENCH Q6_K 2560x512 T=2 multi 4.1 us | il rows1 6.4 rows2 7.4 rows4 6.7 us
BENCH Q6_K 2560x512 T=3 multi 3.9 us | il rows1 6.3 rows2 4.5 rows4 5.4 us
BENCH Q6_K 2560x512 T=4 multi 4.1 us | il rows1 6.5 rows2 4.5 rows4 5.9 us
BENCH IQ4_XS 2560x12288 T=1 multi 12.8 us | il rows1 12.5 rows2 12.3 rows4 12.9 us
BENCH IQ4_XS 2560x12288 T=2 multi 15.0 us | il rows1 13.3 rows2 13.4 rows4 13.8 us
BENCH IQ4_XS 2560x12288 T=3 multi 18.2 us | il rows1 16.1 rows2 14.6 rows4 14.0 us
BENCH IQ4_XS 2560x12288 T=4 multi 22.4 us | il rows1 15.4 rows2 14.2 rows4 14.3 us
BENCH IQ4_XS 2560x640 T=1 multi 3.0 us | il rows1 3.2 rows2 3.2 rows4 5.1 us
BENCH IQ4_XS 2560x640 T=2 multi 3.4 us | il rows1 3.5 rows2 3.7 rows4 3.5 us
BENCH IQ4_XS 2560x640 T=3 multi 3.5 us | il rows1 5.9 rows2 5.1 rows4 3.5 us
BENCH IQ4_XS 2560x640 T=4 multi 3.8 us | il rows1 4.7 rows2 3.7 rows4 3.7 us
BENCH Q4_K 2560x6144 T=1 multi 7.8 us | il rows1 7.3 rows2 7.5 rows4 10.6 us
BENCH Q4_K 2560x6144 T=2 multi 10.2 us | il rows1 8.6 rows2 9.3 rows4 16.5 us
BENCH Q4_K 2560x6144 T=3 multi 11.7 us | il rows1 8.9 rows2 8.4 rows4 8.5 us
BENCH Q4_K 2560x6144 T=4 multi 14.0 us | il rows1 10.3 rows2 8.9 rows4 14.5 us
BENCH Q4_K 2560x640 T=1 multi 2.9 us | il rows1 3.3 rows2 3.0 rows4 5.5 us
BENCH Q4_K 2560x640 T=2 multi 3.4 us | il rows1 5.1 rows2 5.0 rows4 8.2 us
BENCH Q4_K 2560x640 T=3 multi 3.5 us | il rows1 4.6 rows2 3.5 rows4 4.0 us
BENCH Q4_K 2560x640 T=4 multi 3.5 us | il rows1 4.2 rows2 3.6 rows4 7.9 us
BENCH Q4_K 2560x512 T=1 multi 3.1 us | il rows1 3.3 rows2 3.4 rows4 5.5 us
BENCH Q4_K 2560x512 T=2 multi 3.6 us | il rows1 4.8 rows2 4.9 rows4 8.2 us
BENCH Q4_K 2560x512 T=3 multi 3.5 us | il rows1 4.5 rows2 3.6 rows4 4.0 us
BENCH Q4_K 2560x512 T=4 multi 4.0 us | il rows1 4.0 rows2 3.6 rows4 7.9 us
BENCH Q5_K 2560x10240 T=1 multi 13.5 us | il rows1 13.1 rows2 13.0 rows4 13.8 us
BENCH Q5_K 2560x10240 T=2 multi 16.9 us | il rows1 15.4 rows2 14.2 rows4 14.0 us
BENCH Q5_K 2560x10240 T=3 multi 19.8 us | il rows1 15.8 rows2 15.8 rows4 14.5 us
BENCH Q5_K 2560x10240 T=4 multi 23.5 us | il rows1 16.6 rows2 15.9 rows4 18.5 us
BENCH Q5_K 2560x640 T=1 multi 3.3 us | il rows1 3.3 rows2 4.4 rows4 5.7 us
BENCH Q5_K 2560x640 T=2 multi 3.4 us | il rows1 4.2 rows2 4.7 rows4 4.9 us
BENCH Q5_K 2560x640 T=3 multi 3.9 us | il rows1 4.2 rows2 4.0 rows4 4.3 us
BENCH Q5_K 2560x640 T=4 multi 3.6 us | il rows1 4.4 rows2 3.8 rows4 8.4 us
BENCH Q5_K 2560x512 T=1 multi 3.0 us | il rows1 3.1 rows2 4.4 rows4 5.7 us
BENCH Q5_K 2560x512 T=2 multi 3.5 us | il rows1 3.9 rows2 4.6 rows4 4.8 us
BENCH Q5_K 2560x512 T=3 multi 3.9 us | il rows1 4.0 rows2 3.9 rows4 4.2 us
BENCH Q5_K 2560x512 T=4 multi 3.9 us | il rows1 4.0 rows2 3.7 rows4 8.4 us
BENCH Q6_K 2560x6144 T=1 multi 10.6 us | il rows1 10.8 rows2 15.8 rows4 26.6 us
BENCH Q6_K 2560x6144 T=2 multi 13.0 us | il rows1 11.3 rows2 13.7 rows4 13.8 us
BENCH Q6_K 2560x6144 T=3 multi 14.6 us | il rows1 11.7 rows2 11.3 rows4 11.7 us
BENCH Q6_K 2560x6144 T=4 multi 17.0 us | il rows1 15.6 rows2 11.5 rows4 12.9 us
BENCH Q6_K 2560x640 T=1 multi 3.3 us | il rows1 5.1 rows2 7.7 rows4 14.1 us
BENCH Q6_K 2560x640 T=2 multi 3.6 us | il rows1 6.4 rows2 7.4 rows4 6.7 us
BENCH Q6_K 2560x640 T=3 multi 3.8 us | il rows1 6.8 rows2 4.8 rows4 5.5 us
BENCH Q6_K 2560x640 T=4 multi 4.3 us | il rows1 6.9 rows2 4.7 rows4 6.0 us
BENCH Q3_K 2560x10240 T=1 multi 13.6 us | il rows1 11.7 rows2 10.2 rows4 12.4 us
BENCH Q3_K 2560x10240 T=2 multi 18.9 us | il rows1 12.8 rows2 10.9 rows4 12.1 us
BENCH Q3_K 2560x10240 T=3 multi 21.5 us | il rows1 15.8 rows2 13.6 rows4 12.8関連リンク
インストール・モデル・リリースへの站内リンク。