Pull requests / #1774

#1774 sycl opt-in: STRATA_IQ3S_WIDE=1, a wide mmvq kernel for the dense IQ3_S matrices

open · @recutita · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux

Beschreibung

## Title
Issue: none.

## Summary
An opt-in wide kernel for the dense IQ3_S matrices on the SYCL port: `STRATA_IQ3S_WIDE=1`. Without it nothing changes (same kernel, same output as main).

## What changed
- `sycl/src/kernels/cuda/iq_kernels.dp.cpp`: `iq3s_wide_kernel` and its launcher; `iq_mmvq` takes it for IQ3_S (type 21) when `STRATA_IQ3S_WIDE` is set and not 0, and the row length is a multiple of 256. It has the shape of the port's Q4_K..IQ4_XS wide kernels: 8 lanes share a 256-block, lane g takes sub-block g, so a 32-lane sub-group has four blocks in flight; one row per sub-group. It is instantiated for 1 to 8 columns, like the multi kernel.
- Each sub-block's value is `vec_dot_iq3_s_q8_1`'s (same integer sum, same `d * ds * sumi`). Only the order the sub-blocks are summed in differs from `mmvq_multi_kernel`, so the output can differ in the last bits.
- `docs/INTEL.md`: the option, the kernel-level numbers below and where the output parts from the default, next to the other switches of the decode kernels (`STRATA_GR_DOWN_SLICED=0`, `STRATA_MMVQ_WIDE_K=0`).

Why: the multi kernel reads these matrices (attn_q, attn_gate, ssm_out, attn_k, the shared experts' gate/up) at 76-114 GB/s on the B70, while the wide kernels of the other types reach 280-320 GB/s.

## How it was checked
unitrace (`--chrome-kernel-logging`) on an Arc Pro B70 (32 GB, xe, Linux), AOT `bmg-g31`, Flash-Next IQ3_XXS on one card, `--tokens` with a 40-token prompt, greedy. Time per call, and the weight bytes of the matrix (110 bytes per 256 weights) over that time.

T=1 decode (`--spec 2 --mtp-max-t 1 --suffix-draft 0`), 64 tokens, one column:

| matrix (rows x cols) | calls per token | multi | wide |
|---|---:|---:|---:|
| attn_q (12288 x 2560) | 1 | 118.4 us, 114 GB/s | 44.0 us, 307 GB/s |
| attn_gate (6144 x 2560) | 10 | 61.2 us, 110 GB/s | 25.1 us, 269 GB/s |
| ssm_out (2560 x 6144) | 3 | 83.7 us, 81 GB/s | 25.9 us, 261 GB/s |
| shared gate/up (640 x 2560) | 40 | 9.2 us, 76 GB/s | 5.6 us, 127 GB/s |
| attn_k (512 x 2560) | 1 | 7.3 us, 77 GB/s | 4.6 us, 123 GB/s |
| all of them, per token | | 1.36 ms | 0.60 ms |

MTP (`--spec 4 --spec-min-p 0.5 --mtp`), 128 tokens; most calls have 4 columns (the verify window), where the weight bytes are read once for all 4:

| matrix | multi, 4 columns | wide, 4 columns |
|---|---:|---:|
| attn_q | 144.5 us, 94 GB/s | 61.6 us, 219 GB/s |
| attn_gate | 74.5 us, 91 GB/s | 34.5 us, 196 GB/s |
| ssm_out | 99.9 us, 68 GB/s | 36.2 us, 187 GB/s |
| shared gate/up | 11.2 us, 63 GB/s | 7.4 us, 95 GB/s |
| attn_k | 9.1 us, 62 GB/s | 6.9 us, 82 GB/s |

The numbers below are on top of #1798 (expert parallelism with a second GPU). On two cards with it, this branch (and the other opt-in, `STRATA_GR_FUSE_NORM=1`, #1775) applied on top, end to end with `--tokens` (the same 40-token prompt, greedy, 32K context), the options switched by environment in one binary, interleaved; T=1 256 tokens, median of 4 runs, MTP 512 tokens, mean of 2:

| | T=1 | MTP |
|---|---:|---:|
| off | 46.06 tok/s | 108.48 tok/s |
| `STRATA_IQ3S_WIDE=1` | 47.68 tok/s (+3.5%) | 111.23 tok/s (+2.5%) |
| `STRATA_GR_FUSE_NORM=1` | 46.92 tok/s (+1.9%) | 109.69 tok/s (output differs) |
| both | 48.60 tok/s (+5.5%) | 112.48 tok/s (+3.7%) |

Output:
- Off (one card): the same tokens as main's binary, T=1 (256 tokens) and MTP (512 tokens), identical output hashes.
- On (one card): the greedy tokens part from the default at token 198 of 256 at T=1; the continuation is ordinary code.
- With `STRATA_IQ3S_WIDE=1` and with both options the tokens on two cards were the same as off for this prompt (T=1 and MTP); with `STRATA_GR_FUSE_NORM=1` alone they part from off, and with MTP the accepted drafts change with the text, so that MTP figure is not a like-for-like speed comparison.

## Extra Notes
- The whole diff was reviewed by a human (the author).
- Only the SYCL port is touched; CUDA and HIP are unchanged. The port's `strata` target was built (AOT `bmg-g31`).
- An earlier version of this branch also had a codebook staged in shared local memory and 2 or 4 rows per sub-group. Neither was faster than this kernel at T=1 on the B70, so they were removed.
- The traces were taken with the MtpDrafter exit fix (#1642) applied: without it a run without `--mtp` crashes at exit and unitrace writes no trace. The kernel is the same.
- Tested only on the B70 under xe on Linux; not on an Arc A-series card (i915) or on Windows.

Assisted-by: Claude Code with Opus 5.5 medium

Mehr auf der Site

Links zu Install, Modellen, Releases.