Pull requests / #1775
#1775 sycl opt-in: STRATA_GR_FUSE_NORM=1, the hyper-connection read in 3 kernels instead of 4
open · @recutita · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux
Description
## Title Issue: none. ## Summary An opt-in way to run the SYCL port's hyper-connection read in 3 kernels instead of 4: `STRATA_GR_FUSE_NORM=1`. Without it nothing changes (same kernels, same output as main). ## What changed - `sycl/src/kernels/cuda/fused_gr.dp.cpp`, sliced path of `fused_gr_read_multi` (the default, `STRATA_GR_DOWN_SLICED`): with the option the norm kernel is not launched. `gr_down_raw_sliced_kernel` takes the down dot products against the unscaled rows and writes each slice's sum of squares; `gr_down_reduce_rs_kernel` computes `rs` from those, writes it for the up kernel, and sums `rs[c] * partial` over the slices in the fixed slice order. `xn` is not written (nothing reads it after the read). - Why it is the same formula: a slice lies in one stream `c`, and the norm scales all of stream `c` by one factor `rs[c]`, so a slice's partial is `rs[c]` times the dot over the unscaled rows. The rounding order differs (the sums of squares are summed per slice, `rs` is applied after the dot), so the output can differ in the last bits. - `docs/INTEL.md`: the option, the kernel-level numbers below and where the output parts from the default, next to `STRATA_GR_DOWN_SLICED=0` and the other switches of the decode kernels. Why: the norm kernel and its barriers sit in front of the weight reads. A read is 97 calls per token and reads 13 MB of bf16 weights per call. A variant that kept the norm's exact order inside the down kernel was slower with MTP and was dropped. ## How it was checked unitrace (`--chrome-kernel-logging`) on an Arc Pro B70 (32 GB, xe, Linux), AOT `bmg-g31`, Flash-Next IQ3_XXS on one card, `--tokens` with a 40-token prompt, greedy. Median time per call of each kernel of one read: | kernel | T=1, off | T=1, on | MTP, off | MTP, on | |---|---:|---:|---:|---:| | `gr_norm_split` | 6.88 us | unused | 7.19 us | unused | | `gr_down_sliced` / `gr_down_raw_sliced` | 16.88 us | 19.48 us | 19.38 us | 22.50 us | | `gr_down_reduce` / `gr_down_reduce_rs` | 3.96 us | 4.27 us | 3.96 us | 4.58 us | | `gr_up_multi` | 19.69 us | 19.69 us | 27.81 us | 26.67 us | | **one read** | 47.4 us | **43.4 us** | 58.3 us | **53.8 us** | | **97 reads** (a token at T=1, a verify round with MTP) | 4.60 ms | **4.21 ms** | 5.66 ms | **5.21 ms** | T=1 is `--spec 2 --mtp-max-t 1 --suffix-draft 0` (64 tokens), MTP is `--spec 4 --spec-min-p 0.5 --mtp` (128 tokens). The down kernel takes 2.6-3.1 us longer and the reduce kernel 0.3-0.6 us longer, less than the norm kernel they replace (6.9-7.2 us). The numbers below are on top of #1798 (expert parallelism with a second GPU). On two cards with it, this branch (and the other opt-in, `STRATA_IQ3S_WIDE=1`, #1774) applied on top, end to end with `--tokens` (the same 40-token prompt, greedy, 32K context), the options switched by environment in one binary, interleaved; T=1 256 tokens, median of 4 runs, MTP 512 tokens, mean of 2: | | T=1 | MTP | |---|---:|---:| | off | 46.06 tok/s | 108.48 tok/s | | `STRATA_IQ3S_WIDE=1` | 47.68 tok/s (+3.5%) | 111.23 tok/s (+2.5%) | | `STRATA_GR_FUSE_NORM=1` | 46.92 tok/s (+1.9%) | 109.69 tok/s (output differs) | | both | 48.60 tok/s (+5.5%) | 112.48 tok/s (+3.7%) | Output: - Off (one card): the same tokens as main's binary, T=1 (256 tokens) and MTP (512 tokens), identical output hashes. - On (one card): the greedy tokens part from the default at token 198 of 256 at T=1; the continuation is ordinary code. - With `STRATA_IQ3S_WIDE=1` and with both options the tokens on two cards were the same as off for this prompt (T=1 and MTP); with `STRATA_GR_FUSE_NORM=1` alone they part from off, and with MTP the accepted drafts change with the text, so that MTP figure is not a like-for-like speed comparison. ## Extra Notes - The whole diff was reviewed by a human (the author). - Only the SYCL port is touched; CUDA and HIP are unchanged. The port's build was checked (AOT `bmg-g31`): all targets including the parity tests, and `strata` and `gr_parity` again after the last change (a `static_assert`). `gr_parity` with `STRATA_GR_FUSE_NORM=1` prints the same as without it and as main: its fused-read check (a window against single-token calls, direct and graph replays) stays bitwise; its reference checks run `gr_read`, which this does not touch; its one failure (STRATA_QFUSE, not supported on the port) is the same on main. - The traces were taken with the MtpDrafter exit fix (#1642) applied: without it a run without `--mtp` crashes at exit and unitrace writes no trace. The kernels are the same. - Tested only on the B70 under xe on Linux; not on an Arc A-series card (i915) or on Windows. Assisted-by: Claude Code with Opus 5.5 medium
Sur le site
Liens install, modèles, releases.