贡献 / #2089
#2089 CUDA sm_75: interleaved 2-4 column GEMV with a Turing rows table (CMP 50HX), opt-in STRATA_SM75_TABLE=1
closed · @xrip · 0 评论 · 去 GitHub 看
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsDocumentationLinux
说明
## What A rows-a-warp table for **Turing (sm_75)** in `native_mmvq.cu` (`kIlRows_75` for 2-4 columns, `kIl1Rows_75` for one column), measured with `mmvq_il_parity --bench --emit-table` as `docs/MMVQ_IL_TABLE.md` asks. Opt-in with `STRATA_SM75_TABLE=1`, like Volta's, until a second Turing card confirms it. Without the variable nothing changes. ## Why Turing had no table, so the interleaved path was off there and the plain kernels ran. Turing has dp4a, and on this card the interleaved kernels are much faster at 2-4 columns, which is every verify window with `--spec`. The RTX 20 series and the T4 are sm_75 as well. ## Measured **Card:** NVIDIA CMP 50HX, a TU102 at RTX 2080 Ti clocks (2100 MHz SM, 56 SMs). Driver 610.43.03, CUDA 13.3, GCC 14, Linux. **Parity:** `mmvq_il_parity: OK (every case, T 1-4, rows 1/2/4 and the table, bitwise)`. **Selected shapes from `--bench`** (the full output is below): | shape | columns | plain | interleaved, best rows | |---|---:|---:|---:| | IQ4_XS 2560x10240 | 4 | 94.6 us | 35.6 us | | Q4_K 2560x10240 | 4 | 88.3 us | 45.3 us | | Q6_K 2560x12288 | 4 | 123.6 us | 70.5 us | | Q5_K head 2560x248320 | 3 | 1781.2 us | 1097.3 us | | Q5_K head 2560x248320 | 1 | 954.3 us | 764.7 us | **Decode A/B:** - **Setup:** 2x CMP 50HX, `--layer-split auto`, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, `--spec 4 --mtp`, `--kv int8`, `--adapt-swaps 0 --pcie-frac 0.17`. - **Method:** one binary, with the arms differing only by the variable. A fixed 35.7K-token prompt, then 6 questions with 512 greedy tokens each (the prefix is reused, so the time measured is decode). Two runs per arm. - **Result:** 72.62 / 72.57 tok/s without the table, 77.85 / 77.46 tok/s with it (**+7.0 %**). The prompt read did not change. On this box, two runs of the same arm already differ late in 1-2 of the 6 answers, so the texts cannot judge bit-exactness here. The parity test does. ## Also measured, not in this PR: expert mode 8 on Turing `STRATA_EXP_MODE=8` (Volta's and the P100's layout) on the same card: - `native_expert_bench` over all 48 layers of the IQ3_XXS pack: **bitwise equal** in every case. - **gate/up:** IQ3_S, IQ2_S, IQ2_XXS and IQ3_XXS are -27..-42 % at 4 tokens. IQ2_XS is +19 % at 1 token. - **down:** Q2_0 is -9..-12 %. IQ4_NL is +31 % at 4 tokens. - **Whole expert call:** -5 % at 1 token, -14 % at 4. - **Decode:** +2.9 % on its own, and +10.3 % together with this table (72.6 -> 80.0 tok/s). I tried a per-type exception (IQ4_NL down kept on the CUDA layout). The bench improved to -17.5 % at 4 tokens, but decode did not change, so I left it out. If you want sm_75 to take mode 8 by default, as the P100 does, that would be a one-line follow-up. ## Testing - `mmvq_il_parity` (parity and bench) on the card. - Decode A/B as above. - Builds for sm_75 (CUDA 13.3, GCC 14). Not run on other Turing cards. <details><summary>Full <code>mmvq_il_parity --bench --emit-table</code> output (CMP 50HX)</summary> ``` mmvq_il_parity: OK (every case, T 1-4, rows 1/2/4 and the table, bitwise) BENCH IQ4_XS 2560x10240 T=1 multi 39.5 us | il rows1 28.9 rows2 28.0 rows4 30.4 us BENCH IQ4_XS 2560x10240 T=2 multi 60.2 us | il rows1 34.1 rows2 32.6 rows4 32.8 us BENCH IQ4_XS 2560x10240 T=3 multi 78.8 us | il rows1 45.1 rows2 35.1 rows4 35.5 us BENCH IQ4_XS 2560x10240 T=4 multi 94.6 us | il rows1 42.8 rows2 35.6 rows4 40.8 us BENCH IQ4_XS 2560x6144 T=1 multi 19.6 us | il rows1 18.8 rows2 18.3 rows4 20.8 us BENCH IQ4_XS 2560x6144 T=2 multi 38.5 us | il rows1 21.6 rows2 20.3 rows4 21.8 us BENCH IQ4_XS 2560x6144 T=3 multi 46.2 us | il rows1 28.2 rows2 22.8 rows4 23.7 us BENCH IQ4_XS 2560x6144 T=4 multi 58.1 us | il rows1 27.8 rows2 23.3 rows4 26.6 us BENCH IQ4_XS 6144x2560 T=1 multi 19.9 us | il rows1 18.8 rows2 18.2 rows4 19.0 us BENCH IQ4_XS 6144x2560 T=2 multi 31.3 us | il rows1 22.9 rows2 21.7 rows4 20.7 us BENCH IQ4_XS 6144x2560 T=3 multi 40.0 us | il rows1 27.2 rows2 25.3 rows4 22.6 us BENCH IQ4_XS 6144x2560 T=4 multi 49.2 us | il rows1 25.0 rows2 20.1 rows4 26.1 us BENCH IQ4_XS 2560x512 T=1 multi 4.3 us | il rows1 4.6 rows2 4.7 rows4 6.8 us BENCH IQ4_XS 2560x512 T=2 multi 5.0 us | il rows1 5.0 rows2 4.9 rows4 6.1 us BENCH IQ4_XS 2560x512 T=3 multi 6.1 us | il rows1 5.7 rows2 5.6 rows4 5.5 us BENCH IQ4_XS 2560x512 T=4 multi 7.2 us | il rows1 5.2 rows2 4.9 rows4 5.9 us BENCH Q4_K 2560x10240 T=1 multi 38.8 us | il rows1 28.6 rows2 28.5 rows4 32.0 us BENCH Q4_K 2560x10240 T=2 multi 53.5 us | il rows1 59.2 rows2 38.2 rows4 32.3 us BENCH Q4_K 2560x10240 T=3 multi 72.0 us | il rows1 62.7 rows2 41.1 rows4 45.0 us BENCH Q4_K 2560x10240 T=4 multi 88.3 us | il rows1 63.1 rows2 45.3 rows4 59.1 us BENCH Q4_K 6144x2560 T=1 multi 17.7 us | il rows1 19.4 rows2 18.1 rows4 22.8 us BENCH Q4_K 6144x2560 T=2 multi 30.7 us | il rows1 32.4 rows2 23.1 rows4 20.8 us BENCH Q4_K 6144x2560 T=3 multi 40.6 us | il rows1 39.3 rows2 26.9 rows4 32.0 us BENCH Q4_K 6144x2560 T=4 multi 49.1 us | il rows1 42.1 rows2 28.6 rows4 34.2 us BENCH Q5_K 2560x12288 T=1 multi 49.2 us | il rows1 40.1 rows2 40.5 rows4 56.9 us BENCH Q5_K 2560x12288 T=2 multi 70.9 us | il rows1 70.8 rows2 49.2 rows4 47.3 us BENCH Q5_K 2560x12288 T=3 multi 94.9 us | il rows1 78.7 rows2 55.0 rows4 54.5 us BENCH Q5_K 2560x12288 T=4 multi 110.6 us | il rows1 81.3 rows2 66.9 rows4 71.9 us BENCH Q5_K 2560x6144 T=1 multi 25.6 us | il rows1 21.6 rows2 21.6 rows4 32.3 us BENCH Q5_K 2560x6144 T=2 multi 37.8 us | il rows1 34.0 rows2 27.2 rows4 25.5 us BENCH Q5_K 2560x6144 T=3 multi 47.7 us | il rows1 41.7 rows2 29.2 rows4 30.4 us BENCH Q5_K 2560x6144 T=4 multi 56.6 us | il rows1 42.4 rows2 36.9 rows4 39.5 us BENCH Q5_K 6144x2560 T=1 multi 21.8 us | il rows1 22.2 rows2 21.9 rows4 33.9 us BENCH Q5_K 6144x2560 T=2 multi 32.4 us | il rows1 36.7 rows2 27.0 rows4 25.0 us BENCH Q5_K 6144x2560 T=3 multi 42.3 us | il rows1 43.0 rows2 29.4 rows4 25.7 us BENCH Q5_K 6144x2560 T=4 multi 51.6 us | il rows1 42.3 rows2 40.2 rows4 35.9 us BENCH Q5_K head 2560x248320 T=1 multi 954.3 us | il rows1 899.9 rows2 764.7 rows4 972.5 us BENCH Q5_K head 2560x248320 T=2 multi 1356.3 us | il rows1 1366.3 rows2 1027.9 rows4 963.6 us BENCH Q5_K head 2560x248320 T=3 multi 1781.2 us | il rows1 1530.7 rows2 1212.3 rows4 1097.3 us BENCH Q5_K head 2560x248320 T=4 multi 2178.7 us | il rows1 1627.5 rows2 1327.5 rows4 1398.3 us BENCH Q6_K 2560x12288 T=1 multi 61.4 us | il rows1 49.0 rows2 50.2 rows4 70.6 us BENCH Q6_K 2560x12288 T=2 multi 79.1 us | il rows1 63.4 rows2 55.4 rows4 74.2 us BENCH Q6_K 2560x12288 T=3 multi 102.3 us | il rows1 81.8 rows2 66.6 rows4 64.7 us BENCH Q6_K 2560x12288 T=4 multi 123.6 us | il rows1 85.7 rows2 70.5 rows4 74.5 us BENCH Q6_K 2560x10240 T=1 multi 56.5 us | il rows1 41.5 rows2 41.5 rows4 65.4 us BENCH Q6_K 2560x10240 T=2 multi 65.6 us | il rows1 56.0 rows2 49.4 rows4 69.1 us BENCH Q6_K 2560x10240 T=3 multi 85.3 us | il rows1 68.4 rows2 56.7 rows4 53.3 us BENCH Q6_K 2560x10240 T=4 multi 102.7 us | il rows1 71.4 rows2 60.0 rows4 62.2 us BENCH Q6_K 6144x2560 T=1 multi 27.5 us | il rows1 31.5 rows2 29.5 rows4 52.8 us BENCH Q6_K 6144x2560 T=2 multi 36.7 us | il rows1 40.8 rows2 32.6 rows4 47.8 us BENCH Q6_K 6144x2560 T=3 multi 47.9 us | il rows1 39.6 rows2 36.0 rows4 35.3 us BENCH Q6_K 6144x2560 T=4 multi 57.9 us | il rows1 45.3 rows2 38.1 rows4 35.8 us BENCH Q6_K 2560x512 T=1 multi 4.9 us | il rows1 5.9 rows2 6.9 rows4 17.2 us BENCH Q6_K 2560x512 T=2 multi 5.4 us | il rows1 7.0 rows2 7.8 rows4 15.3 us BENCH Q6_K 2560x512 T=3 multi 6.1 us | il rows1 6.0 rows2 8.5 rows4 9.2 us BENCH Q6_K 2560x512 T=4 multi 7.0 us | il rows1 7.1 rows2 9.0 rows4 9.9 us BENCH IQ4_XS 2560x12288 T=1 multi 51.8 us | il rows1 34.0 rows2 32.8 rows4 37.2 us BENCH IQ4_XS 2560x12288 T=2 multi 71.1 us | il rows1 41.0 rows2 38.7 rows4 39.4 us BENCH IQ4_XS 2560x12288 T=3 multi 92.1 us | il rows1 50.9 rows2 40.7 rows4 41.5 us BENCH IQ4_XS 2560x12288 T=4 multi 112.9 us | il rows1 51.2 rows2 43.2 rows4 49.0 us BENCH IQ4_XS 2560x640 T=1 multi 4.6 us | il rows1 4.6 rows2 4.7 rows4 7.0 us BENCH IQ4_XS 2560x640 T=2 multi 5.8 us | il rows1 5.3 rows2 5.1 rows4 6.5 us BENCH IQ4_XS 2560x640 T=3 multi 7.1 us | il rows1 5.8 rows2 5.6 rows4 5.8 us BENCH IQ4_XS 2560x640 T=4 multi 8.9 us | il rows1 5.5 rows2 5.0 rows4 6.2 us BENCH Q4_K 2560x6144 T=1 multi 21.3 us | il rows1 18.4 rows2 18.1 rows4 19.1 us BENCH Q4_K 2560x6144 T=2 multi 35.3 us | il rows1 33.3 rows2 26.0 rows4 22.3 us BENCH Q4_K 2560x6144 T=3 multi 44.7 us | il rows1 39.9 rows2 27.1 rows4 29.8 us BENCH Q4_K 2560x6144 T=4 multi 54.3 us | il rows1 39.5 rows2 28.4 rows4 39.8 us BENCH Q4_K 2560x640 T=1 multi 4.9 us | il rows1 4.8 rows2 5.3 rows4 7.7 us BENCH Q4_K 2560x640 T=2 multi 5.7 us | il rows1 6.3 rows2 6.0 rows4 6.4 us BENCH Q4_K 2560x640 T=3 multi 7.1 us | il rows1 5.8 rows2 5.8 rows4 10.6 us BENCH Q4_K 2560x640 T=4 multi 7.8 us | il rows1 5.8 rows2 6.0 rows4 10.8 us BENCH Q4_K 2560x512 T=1 multi 4.0 us | il rows1 4.4 rows2 4.4 rows4 6.7 us BENCH Q4_K 2560x512 T=2 multi 5.1 us | il rows1 5.0 rows2 5.0 rows4 6.1 us BENCH Q4_K 2560x512 T=3 multi 5.7 us | il rows1 5.1 rows2 5.5 rows4 9.9 us BENCH Q4_K 2560x512 T=4 multi 6.6 us | il rows1 5.2 rows2 5.7 rows4 10.3 us BENCH Q5_K 2560x10240 T=1 multi 43.4 us | il rows1 33.9 rows2 34.2 rows4 43.8 us BENCH Q5_K 2560x10240 T=2 multi 60.8 us | il rows1 63.4 rows2 43.0 rows4 39.5 us BENCH Q5_K 2560x10240 T=3 multi 79.4 us | il rows1 67.3 rows2 46.4 rows4 45.5 us BENCH Q5_K 2560x10240 T=4 multi 93.8 us | il rows1 68.5 rows2 57.2 rows4 60.8 us BENCH Q5_K 2560x640 T=1 multi 5.1 us | il rows1 4.9 rows2 5.5 rows4 12.3 us BENCH Q5_K 2560x640 T=2 multi 5.9 us | il rows1 6.1 rows2 6.1 rows4 7.8 us BENCH Q5_K 2560x640 T=3 multi 7.3 us | il rows1 6.1 rows2 6.1 rows4 7.7 us BENCH Q5_K 2560x640 T=4 multi 8.2 us | il rows1 6.2 rows2 8.0 rows4 10.7 us BENCH Q5_K 2560x512 T=1 multi 4.2 us | il rows1 4.5 rows2 4.7 rows4 11.2 us BENCH Q5_K 2560x512 T=2 multi 5.4 us | il rows1 5.0 rows2 5.3 rows4 7.2 us BENCH Q5_K 2560x512 T=3 multi 6.3 us | il rows1 5.3 rows2 5.7 rows4 7.1 us BENCH Q5_K 2560x512 T=4 multi 7.0 us | il rows1 5.4 rows2 7.4 rows4 10.3 us BENCH Q6_K 2560x6144 T=1 multi 31.7 us | il rows1 26.7 rows2 26.1 rows4 36.0 us BENCH Q6_K 2560x6144 T=2 multi 41.7 us | il rows1 37.7 rows2 30.7 rows4 45.8 us BENCH Q6_K 2560x6144 T=3 multi 51.3 us | il rows1 42.0 rows2 37.4 rows4 34.2 us BENCH Q6_K 2560x6144 T=4 multi 62.4 us | il rows1 44.5 rows2 40.3 rows4 40.2 us BENCH Q6_K 2560x640 T=1 multi 5.7 us | il rows1 6.5 rows2 7.4 rows4 18.8 us BENCH Q6_K 2560x640 T=2 multi 6.3 us | il rows1 7.2 rows2 8.1 rows4 15.7 us BENCH Q6_K 2560x640 T=3 multi 7.1 us | il rows1 6.6 rows2 8.9 rows4 9.4 us BENCH Q6_K 2560x640 T=4 multi 8.4 us | il rows1 7.6 rows2 9.5 rows4 10.3 us BENCH Q3_K 2560x10240 T=1 multi 53.7 us | il rows1 39.0 rows2 34.3 rows4 51.8 us BENCH Q3_K 2560x10240 T
本站相关内容
相关页面的快捷入口。