Pull requests / #2810
#2810 sycl: STRATA_HC_Q8=1 reads the Q8_0 hyper-connection projections (it loaded them and never read them)
open · @hotschmoe · 0 comentários · No GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descrição
## Title sycl: STRATA_HC_Q8=1 reads the Q8_0 hyper-connection projections (it loaded them and never read them) Issue: Resolves #2809 ## Summary On the SYCL (Intel) build `STRATA_HC_Q8=1` did nothing. The loader kept a Q8_0 copy of the hyper-connection projections only when the GGUF stored them as Q8_0 (the ISTA files store BF16, so it copied nothing), and the port's `fused_gr_read_multi` had no Q8_0 read: it ignored `q8_down` / `q8_up` and read the BF16 copies whatever the flag said (`fused_gr.dp.cpp`: "the Q8_0 reads (S23 STRATA_HC_Q8) are not built"). For a GGUF that stores them as Q8_0 the copies were made and never read. (From the source; on a development tree with the loader half fixed, a GPU trace showed the same BF16 kernels with the flag on and off.) With this change the flag does on SYCL what it does on CUDA: the hyper-connection down / up projections (the layers' and the final mixer's) are read as Q8_0 in the verify window - the GGUF's own Q8_0 blocks, or Q8_0 of its BF16 values quantized at load. Flag unset: nothing changes (the pointers stay null and the BF16 read runs as before; outputs and draft counts identical to `main`, checked). As on CUDA (`fused_gr.cu`, "an output-changing step") the Q8_0 read is NOT bitwise the BF16 read: Q8_0-rounded weights, another summation order. Bitwise: the default path; R_out (the pending write) between the two reads; a captured graph's replays against the direct call; the fallback for arguments not at the callers' fixed strides. VRAM, said plainly: on this port the BF16 copies stay in the arena (the prompt path's GEMMs read them), so the flag ADDS 0.63 GiB for Flash-Next (0.40 + 0.24 GiB over a two-card layer split) instead of replacing the BF16 bytes as on a single CUDA GPU. That is cache the experts lose: on 2x B70 the model that ran at `--max-context 65536` without the flag refused 310 experts with it (132 at 32768 with the default 1,024 MiB reserve); the A/B below ran at 32768 with `--vram-reserve-mib 512`. Dropping the BF16 copies (`add_hc_skip` / `hc_bf16_view`) is not ported here. Measured, 2x Intel Arc Pro B70 32 GB (kernel 7.1, Compute Runtime 26.22.38646.4, oneAPI 2026.1, AOT bmg-g31), upstream main 296b59d2 vs this branch, Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS (BF16 hyper-connection tensors), `--layer-split 30 --prefill 8192 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts --expert-cache auto --max-context 32768 --vram-reserve-mib 512`, one slot, greedy, a fixed prompt with a 512-token coding reply, 3 runs, same binary, flag unset vs set: - verify window 39.4 / 39.6 / 39.8 -> 37.4 / 37.5 / 37.4 ms (-5.3%; independent of draft acceptance); - decode 81.8 / 79.4 / 78.5 -> 85.0 / 88.0 / 86.0 tok/s (median 79.4 -> 86.0, +8.3%; part of it is a different draft acceptance of the changed output: drafts accepted 353/474, 349/489, 348/492 -> 351/489, 357/471, 353/483); - prompt reading unchanged: 4K 1,093 / 1,190 -> 1,093 / 1,188 tok/s, 16K 1,774 / 1,749 -> 1,769 / 1,750; - quality: teacher-forced NLL of 4,762 tokens of three texts through the decode windows, same targets: 3.1974 unset, 3.1699 set. No degradation measured on this gate; not claimed as an improvement (rounding-only variants move this kind of gate by 0.01-0.1 nats per token). - kernel level (one card, graph replay, random weights not L2-resident, us per read, BF16 -> Q8_0): T = 1 38.7 -> 24.7, T = 4 50.8 -> 29.3, T = 6 62.0 -> 34.4. - `gr_q8_parity --selftest`: 33 of 33 cases pass. Coherence and per-card health after every run: pass. ## What changed - `sycl/src/kernels/cuda/fused_gr.dp.cpp`: the Q8_0 read (`namespace gr_q8`), two kernels, one row per 16-lane sub-group lane. The down kernel stages a 512-column slab of `R' * w_norm` (unscaled) for the T rows with the slab's sums of squares; each lane holds 4 rows' quants and scales over its sub-group's 32 columns; the 16 sub-groups add their partials in local memory in a fixed order; the LAST work-group of a row group to arrive (an atomic counter, no waiting, so nothing assumes co-residency and it is safe inside a SYCL graph) applies rs per stream, writes lo / the injection and rs, and resets the counter. The up kernel splits each row's K = 320 over five sub-groups, sums the fifths in a fixed order, and runs the default read's epilogue (pending write, gate, stream mean). `w = d * q` is exact in FP32. The inject rows are read from the pack's BF16, as the CUDA read's default. One guarded line in `fused_gr_read_multi`, first thing after the queue is known: `if (a[0].q8_down && a[0].q8_up && gr_q8::read(m, st)) return false;` - null pointers (the default) skip it. Arguments whose rows are not at the fixed strides every caller uses (verify, mtp, the final mixer) fall back to the BF16 read with one line on stderr. The scratch (partials, sums of squares, counters) is one per queue, allocated and zeroed outside any capture. `fused_gr_q8_0_split_host`: the read's weight layout - every row's int8 quants row-major, then every row's fp16 scales; the same bytes as ggml's 34-byte blocks, reordered. - `sycl/include/strata/kernels/fused_gr_q8.hpp` (new, the port's own include tree): declares that layout function for the loader and the parity test. - `sycl/src/core/native_dense.cpp`: with `STRATA_HC_Q8=1`, hyper-connection tensors of GGUF type 8 (Q8_0, as they are) AND type 30 (BF16: quantized on the host with the file's existing `q8_0_of`, ggml's rounding `d = amax / 127` as fp16, `q = round(x / d)`) are uploaded in the split layout. Shape-checked against the compiled geometry (hc_lr x hc * n_embd or the transpose); another stage's layers are skipped in a layer split (`outside` / `in_range`, as the main upload loop does). The inject rows are no longer copied: nothing read them before and the read takes the BF16 rows, so `STRATA_HC_Q8_INJECT` has no effect on SYCL (said in the comment). - `sycl/src/kernels/gr_q8_parity.cpp` (new) + `sycl/CMakeLists.txt` (one name in the `--selftest` loop): real geometry, T = 1..8 x pending write on / off x inject rows on / off. The Q8_0 read against an FP64 reference computed from the very Q8_0 bytes it reads (lo / rs / inject / mixed within 1e-4 of the largest value - FP32 summation order), R_out bitwise the BF16 read's, a captured graph replayed twice bitwise the direct call, the Q8_0 read's `mixed` different from the BF16 read's by the weights' rounding (proves the Q8_0 bytes are read - the bug), and the fixed-stride fallback bitwise the BF16 read. - `docs/INTEL.md`: a section on the flag: what was wrong, what it does now, what is and is not bitwise, the VRAM cost, the numbers. Not changed: anything under `src/` or `include/` (CUDA / HIP untouched); the default SYCL read and every other path (flag unset: the new code is never entered); `STRATA_HC_Q8`'s meaning on CUDA / HIP / Mac; `STRATA_GR_V3`, `STRATA_HC_SPLIT` and the port's other hyper-connection switches. ## Extra Notes - Built: the SYCL engine and `gr_q8_parity` (oneAPI 2026.1, AOT bmg-g31). CUDA and HIP are not touched by any file of this PR (only `sycl/**` and `docs/INTEL.md`). - The kernel-level numbers come from a probe in a development tree with the same two kernels (graph replay, random BF16 weights and their Q8_0, 8 weight sets, 48 reads per graph); everything else above was measured on this branch against `main`. - Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.