Pull requests / #2150

#2150 SYCL: opt in to 256 GRFs for fused hyperconnection up

open · @alex4o · 0 コメント · GitHub で見る

BenchmarksAMD / HIPNVIDIA / CUDAModels & quants

本文

## Summary

Add an opt-in 256-GRF launch for the default fused hyperconnection up kernel. On an Intel Arc A770 16 GB / i915 with oneAPI 2026.1, this removed the measured register-spill allocation and improved matched warmed decode by about 13%.

Related: #1606 (decode optimization only; this does not claim to fix that report's prefill bottleneck).

## What changed

Only sycl/src/kernels/cuda/fused_gr.dp.cpp changes:

- STRATA_GR_UP_LARGE_GRF=1 selects grf_size<256> and a distinct kernel identity.
- Unset, 0, or any value other than 1 retains the original kernel name and launch properties.
- Body, arithmetic, indexing, range, subgroup size 32, and root-sync property are shared/unchanged. Other GR paths are untouched.
- The flag is sampled once: restart the engine to change it.

The option remains off by default. The experimental Intel GRF header is an unconditional compile dependency; validation is on oneAPI 2026.1, the toolchain documented for this port. Older compilers and other GPUs were not tested.

## Validation

**This PR on base a0116b2:** full SYCL strata and gr_bench builds passed in strata-sycl-dev with icpx 2026.1. Separate gr_bench runs printed effective flag values unset / 0 / 1; all six output checksums (T=1..6) matched each other and the earlier OFF checksums. CUDA/HIP source is unchanged.

**Earlier full-model experiments on base 61b3fb5, with this exact kernel patch:**

| Matched OFF decode | Matched ON decode |
|---:|---:|
| 18.7 tokens/s | 21.2 tokens/s |
| 18.6 tokens/s | 21.1 tokens/s |
| 19.3 tokens/s | 21.4 tokens/s |

Ryzen 5800X, 48 GB-class RAM, IQ3_XXS, INT8 KV, 131072 capacity, 4096 prefill, spec4, CPU cache misses. Two warm-ups per arm; three 384-token timing samples without prefix reuse. Median gain 13.4%. These capped samples are not complete-answer correctness tests.

- Compiler metadata: 3520-byte spill allocation -> zero; 256 GRFs. Private-memory size remains 40192 bytes.
- A direct fused-kernel fixture covered 432 cases / 77,575,968 output floats, including T=1..8, injection/apply/in-place variants. OFF/ON and repeated OFF were bit-identical; finite/guard checks passed.
- A 129,845-token prompt and complete 549-token OFF reference vs exact-token ON replay produced 136,327,680 bit-identical float32 logits. Both arms used the same isolated diagnostic capture/teardown fix and controlled T1/adaptation-off execution; neither diagnostic is part of this PR.
- An independent unforced ON run with production speculation/adaptation emitted the same complete answer. OFF and ON recovered 5/5 codes and passed 384 LRU checks each.
- A later OFF/OFF repeat showed numerical variation. The matched equality is evidence for this tested pair, not a claim of deterministic repetition or universal numerical equivalence.

[Detailed protocol, limitations and reproducible direct fixture](https://gist.github.com/alex4o/6f5912c82d67393fe5ca3aac26ff5b63). Full-model speed/128K tests were not rerun on current upstream a0116b2; the changed GRF source was unchanged upstream between the two bases. No universal 20+ tokens/s claim.

関連リンク

インストール・モデル・リリースへの站内リンク。