Pull requests / #2203
#2203 sycl/A770: fused GR read with more, smaller work-groups on Arc A-series (same sums; decode +3-5%; on top of #2193)
open · @aslater3 · 0 comments · View on GitHub
BenchmarksNVIDIA / CUDAModels & quantsLinux
Description
## Title Depends on #2193 (this branch contains its commit; review only the second commit, `sycl/A770: fused GR read with more, smaller work-groups on Arc A-series`). ## Summary After #2193 the fused GR read on an Arc A770 still launches too few work-groups to fill the card (80 for the sliced down kernel, 160 for the multi-token up kernel). On Alchemist this splits them further: **the same values, summed in the same order**, by more groups. `gr_bench` 1.2-1.3x faster, decode +3-5% on an A770, identical outputs. Default on Alchemist only; every other device launches today's kernels. `STRATA_GR_MORE_GROUPS=1` forces it on, `=0` off. ## What changed `sycl/src/kernels/cuda/fused_gr.dp.cpp` (and one header comment in `sycl/src/kernels/gr_bench.cpp`): - sliced down kernel: one work-group per (slice, row block), 80 x 6 = 480 groups; each row's partial is computed exactly as before, by a different group. - `gr_down_reduce`: 8 slice loads in flight per step, then added one at a time, so the sum accumulates slices 0, 1, ..., 79 in the same sequence of float adds as the old loop (`static_assert(GRS_SLICES % 8 == 0)`; `-fp-model=precise`). - up kernel: templated on the column count, 8 columns per group (320 groups instead of 160) on this path; `UPM_COLS` stays 16 for every other user. Only group 0 writes the per-token `lo` values, which every group computes identically. - `gr_more_groups(queue)`: once per process, `strata::intel_gpu_gen == Alchemist` by default, env override; independent of `STRATA_GR_LARGE_GRF`. ## How it was checked Arc A770 16 GB, `xe`, Linux 7.0, oneAPI 2026.1 (JIT), Ryzen 7 3800X, 30 GB RAM; `a0116b27` + #2190 + #2193 with and without this commit, same day. - `gr_bench`, 1-6 tokens, checksums identical for every token count: | build | us (1 / 2 / 3 / 4 / 5 / 6 tokens) | |---|---| | #2193 | 132.2 / 118.9 / 133.6 / 150.5 / 168.3 / 183.5 | | this PR | 116.1 / 94.6 / 105.8 / 122.9 / 136.4 / 148.4 | - `gr_multi_parity`: 32 cases, 0 failures, with `STRATA_GR_MORE_GROUPS=0` and `=1`. `gr_parity`: only the pre-existing `QFUSE ... not written` failure that `main` has on this card. - Decode, Coder IQ1_M, 8K-token prompt, 256 tokens, greedy, `--resident-experts --max-context 131072 --kv int8 --kv-resident 20480 --spec 4 --mtp`, `STRATA_VERIFY_STEPPED=1`: #2193 22.12 / 22.35 / 22.55 / 22.65 tok/s -> this PR 23.24 / 23.61 / 23.62 tok/s; output ids identical (`72b762d5b2a3`) in all seven runs. Not built or run on Battlemage or i915. SYCL-only file. ## Extra Notes Several other runs of both builds did not finish because of the decode hang reported in #2194; they are left out, not counted against either side. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.