Pull requests / #2193

#2193 sycl: fused GR read without the private GrMulti copy (GR_PICK); large GRF on Arc A-series only - A770 decode 5.4 -> 22 tok/s

open · @aslater3 · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Title
Issue: #1545, #1606 (Arc A770 decode)

## Summary
On an Arc A770 (Alchemist, `xe`) the fused GR read kernels index the by-value kernel argument `GrMulti` with a runtime token index (`m.a[k]`). IGC then copies the whole ~1.2 KB struct into every work-item's private scratch (unitrace: ~40 KB private per SIMD32 thread in `gr_up_multi` / `gr_down_reduce` / `gr_norm_split`). `GR_PICK` selects one field with compile-time indices only, so nothing is copied. Addressing only: the same values, read and summed in the same order, so outputs are bit-identical.

On current `main` this copy is what makes A770 decode crawl with `STRATA_VERIFY_STEPPED=1`: each window segment ends with the fused GR read, and the host waits ~3.5 ms longer per segment. With this PR, 8K decode goes from **5.9 to 22.1-22.6 tok/s with the same output tokens**.

The same commit runs the sliced down kernel and the multi-token up kernel in large-GRF mode (256 registers) on Alchemist only (`strata::intel_gpu_gen`), because at 128 the up kernel still spills ~3.4 KB/thread there. Large GRF halves the threads per Xe core, so Battlemage and every other device launch exactly the kernels they launch today. `STRATA_GR_LARGE_GRF=1` forces it on anywhere, `=0` off.

## What changed
- `sycl/src/kernels/cuda/fused_gr.dp.cpp`: `GR_PICK` (a fold over `std::index_sequence` of selects on the uniform argument) replaces `m.a[k].field` in the multi-token GR kernels; `gr_large_grf(queue)` (once per process, Alchemist by default, env override) picks between the existing launch and a `grf_size<256>` copy of the same kernel for the sliced down and multi-token up launches. On a non-Alchemist device, only the GR_PICK addressing change applies.

## How it was checked
Arc A770 16 GB, `xe`, Linux 7.0, oneAPI 2026.1 (JIT), Ryzen 7 3800X, 30 GB RAM; builds of `a0116b27` (+ #2190, without which this card cannot size its expert cache) with and without this commit, same day.

**Bit-identity.** `gr_bench` (1-6 tokens, real Coder dims) prints the same checksum for every token count on `main`, with this commit, and with `STRATA_GR_LARGE_GRF=0`. `gr_multi_parity`: 32 cases, 0 failures. `gr_parity`: the same single failure as on `main` (`QFUSE: T=1 rep 0: not written` in the fused GR read + q8_1 case) — pre-existing on this card, unrelated to this change.

**Kernel time, `gr_bench`, 1-6 tokens:**

| build | us (1 / 2 / 3 / 4 / 5 / 6 tokens) |
|---|---|
| `main` | 218.0 / 205.5 / 219.4 / 227.7 / 244.9 / 258.4 |
| this PR, `STRATA_GR_LARGE_GRF=0` (GR_PICK only) | 199.6 / 185.5 / 197.4 / 205.3 / 225.0 / 238.8 |
| this PR (Alchemist default: GR_PICK + large GRF) | 132.2 / 118.9 / 133.6 / 150.5 / 168.3 / 183.5 |

**Decode, Coder IQ1_M, 8K-token code prompt, 256 new tokens, greedy**, `--resident-experts --expert-cache auto --max-context 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --prefill-borrow --prefill auto --spec 4 --spec-min-p 0.5 --mtp --adapt-every 8`, `STRATA_VERIFY_STEPPED=1`:

| build | decode tok/s | verify window: wait for rings / pool ms per round |
|---|---|---|
| `main` + #2190 | 5.37; 5.88 and 5.93 with `STRATA_DOORBELL_CHECK=1` / `STRATA_LEND_MIRROR=0` (5 other default runs hung, see the note below) | 69-71 / 183-208 |
| this PR, `STRATA_GR_LARGE_GRF=0` | 20.61, 20.93 | 66.5-67.5 / 35.6-41.2 |
| this PR | 22.12, 22.55, 15.60 (*) | 58.1-59.1 / 33.7-35.2 |

(*) the 15.60 run followed an OOM-killed run (cold page cache; its pool phase was 56.9 ms/round). Output ids are identical to `main` in every run where the expert placement matched (`72b762d5b2a3`); one large-GRF-off run had a different cache placement (hits 0.7605 vs 0.7468) and different ids, which the CPU-vs-GPU expert rounding explains (seen on `main` too).

Not built or run on Battlemage, i915 or Windows; on those the change is the GR_PICK addressing only (and the large-GRF switch if forced). CUDA/HIP: file is SYCL-only.

## Extra Notes
- Follow-ups from the same A770 work, as separate PRs: more work-groups in the GR read (stacked on this one), large GRF for the wide MMVQ kernels, the GDN Xe recurrence, and spill removal in the routers / QSA select / grouped gate-up.
- Separately (issue to follow): on `main` + #2190 this card intermittently hangs at the start of a decode window (main thread in `Verifier::run` -> `queue::wait`, no xe reset logged, RSS grows until the OOM killer); 3 of 5 baseline runs here hit it, 1 of ~15 runs with this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.