Pull requests / #3367

#3367 [Engine] sycl: the hyper-connection read's reduce keeps 8 loads in flight, its up step writes lo once

open · @deriknel · 0 comentários · No GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descrição

## Problem and result
On the Arc Pro B70, the hyper-connection read (`gr`, 2 sites per layer) is one of the largest kernel classes in decode. Its reduce issues one load at a time, and its up step writes `lo` on every iteration. The variants that keep 8 loads in flight and write `lo` once already exist, but only behind `STRATA_GR_MORE_GROUPS`, which bundles them with two layouts that run slower on the B70. This change makes the two variants the default without the slower layouts. Same output bits; decode about 0.7% faster here.

## Related discussion
none. A focused kernel change, no defaults outside these two kernels.

## Approach
`sycl/src/kernels/cuda/fused_gr.dp.cpp`, two launches:
- the down step's reduce launches `gr_down_reduce_kernel<true>`: the same 80 adds in the same order, with 8 loads issued ahead instead of waiting on each L2 load in turn;
- the up step launches `gr_up_multi_kernel<C, false, true>`: the per-token `lo` values are written by block 0 instead of by all 160 blocks (the same values).

Both variants were only reached under `STRATA_GR_MORE_GROUPS` (on by default for the A-series), which also switches in a 480-group down kernel and an 8-column up step; together those lost 2% on two B70s, so they stay where they are. `STRATA_GR_MORE_GROUPS` keeps its meaning.

## Validation
Setup for every run: `main` e88ee39d built for SYCL (AOT `bmg-g31`, oneAPI 2026.1, Compute Runtime 26.35.39758.10, IGC 2.41.5) in a container; host CachyOS, kernel 7.2.11, `xe`. 2x Intel Arc Pro B70 32 GB, layer split 26, Swift 1.5 Flash-Next IQ3_S with every expert in VRAM, `--max-context 262144`, int8 KV, `--spec 4 --spec-min-p 0.6`, `--prefill 4096`. Each build has only this PR's commit on top of `main`. Greedy check: 5 fixed prompts (prose, code, maths, an edit, a story), temperature 0, 400 tokens, thinking off, compared character by character. Pipeline windows off; `main` and this change alternated on fresh engines, two runs each.

**Passed:**
- Decode, bench medians: `main` 83.2 / 82.8 (prose), 106.6 / 106.6 (code); this change 83.7 / 83.7 and 107.4 / 107.3 (+0.8% / +0.7%).
- Greedy text on the 5 prompts identical to `main`'s, both runs.
- `ctest` on both cards: the same 3 failures as on `main` (`gr_parity` among them, failing the same way without this change); `iq_parity` with fixtures, 0 failures.

**Failed:**
- none observed.

**Not run:**
- CUDA, HIP and Metal builds: the change is in `sycl/` only.

## Limits
A small gain. `gr_parity` fails on `main` as well (its `STRATA_QFUSE` case), so it does not check this change.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.

No site

Links install, modelos, releases.