问题 / #2805
#2805 [Feature]: SYCL: decode QSA selection and attention in fewer, shorter launches (opt-in, same bits)
open · @hotschmoe · 0 评论 · 去 GitHub 看
说明
### What do you want to do? Decode Flash-Next faster on an Intel Arc Pro B70 (the SYCL port), with several streams and at long context, without changing a single output bit. With MTP, every verify window runs the QSA selection (block scores, top-k) and the attention for each attention layer. On the B70 these are small, latency-bound launches, and four of them do more serial work than they need to: 1. A batch window (`--batch`, `"parallel"`) runs the scores, top-k and attention one row at a time, because each slot has its own K/V: four launches per row and layer. The kernels already take several rows, each with its own step record. So a slot's rows could share one scores launch, and the whole window could share one top-k launch and one attention launch over every slot's pools. 2. The wide top-k kernel (a capacity above ~135K cells, e.g. `--max-context 200000`) finds each radix pass's digit with one thread walking 256 bins while the other 1,023 wait. 3. The decode block scores run a separate 5-step shuffle reduction for every (query, head) of every block. 4. The attention merge walks every chunk of the capacity, including the chunks with no selected cell. The MTP drafter's 32K window is 512 chunks, one dependent load chain per thread, even at a 1K context. I would like an opt-in switch for each, off by default. Each should give exactly the bits of the path it replaces, checked by a parity test against the existing kernels, so it can be measured and turned on where it pays. Kernel times from a working implementation, one B70 (synthetic layers: 24 q heads, INT8 K/V, a 200,000-cell capacity; graph replay, us): - a window of 2 slots x 2 rows, selection + attention per layer: 184 -> 78 (1K), 365 -> 138 (16K), 498 -> 205 (64K) - the wide top-k, one query: 36.3 -> 19.5 (16K), 57.9 -> 42.1 (64K) - the block scores of four queries: 18.7 -> 14.1 (16K), 63.1 -> 43.2 (64K) - the MTP drafter's attention call: 86.9 -> 41.0 (1K), 460.9 -> 335.9 (64K) Workaround today: none. I have an implementation that touches only `sycl/**` and `docs/INTEL.md`, and will open a pull request that references this issue. Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.
本站相关内容
相关页面的快捷入口。