Issues / #2805

#2805 [Feature]: SYCL: decode QSA selection and attention in fewer, shorter launches (opt-in, same bits)

open · @hotschmoe · 0 comentarios · En GitHub

Models & quantsDocumentation

Descripción

### What do you want to do?

Decode Flash-Next faster on an Intel Arc Pro B70 (the SYCL port), with several streams and at long context, without
changing a single output bit.

With MTP, every verify window runs the QSA selection (block scores, top-k) and the attention for each attention layer.
On the B70 these are small, latency-bound launches, and four of them do more serial work than they need to:

1. A batch window (`--batch`, `"parallel"`) runs the scores, top-k and attention one row at a time, because each slot
   has its own K/V: four launches per row and layer. The kernels already take several rows, each with its own step
   record. So a slot's rows could share one scores launch, and the whole window could share one top-k launch and one
   attention launch over every slot's pools.
2. The wide top-k kernel (a capacity above ~135K cells, e.g. `--max-context 200000`) finds each radix pass's digit
   with one thread walking 256 bins while the other 1,023 wait.
3. The decode block scores run a separate 5-step shuffle reduction for every (query, head) of every block.
4. The attention merge walks every chunk of the capacity, including the chunks with no selected cell. The MTP
   drafter's 32K window is 512 chunks, one dependent load chain per thread, even at a 1K context.

I would like an opt-in switch for each, off by default. Each should give exactly the bits of the path it replaces,
checked by a parity test against the existing kernels, so it can be measured and turned on where it pays.

Kernel times from a working implementation, one B70 (synthetic layers: 24 q heads, INT8 K/V, a 200,000-cell capacity;
graph replay, us):

- a window of 2 slots x 2 rows, selection + attention per layer: 184 -> 78 (1K), 365 -> 138 (16K), 498 -> 205 (64K)
- the wide top-k, one query: 36.3 -> 19.5 (16K), 57.9 -> 42.1 (64K)
- the block scores of four queries: 18.7 -> 14.1 (16K), 63.1 -> 43.2 (64K)
- the MTP drafter's attention call: 86.9 -> 41.0 (1K), 460.9 -> 335.9 (64K)

Workaround today: none.

I have an implementation that touches only `sycl/**` and `docs/INTEL.md`, and will open a pull request that
references this issue.

Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

En el sitio

Enlaces a install, modelos, releases.