#5475 [Engine] opt-in int8 copies for the persistent hc read (STRATA_HC_PERSIST_Q8, STRATA_HC_I8ONLY)

open · @Vop3p · 0 Kommentare · Auf GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Beschreibung

## Problem and result

With `STRATA_HC_PERSIST=1` the verifier's hyper-connection read (down + inject, then up) runs as one persistent kernel per read. On gfx1030 it is bound by memory bandwidth: it streams the bf16 hc weights once per read, and synchronization is only ~4 us of each read. Halving the bytes is the lever. `STRATA_HC_Q8` already makes Q8_0 copies of these weights, but it does not apply to the persistent read: it runs a separate path, which on this card is slower than `HC_PERSIST`.

`STRATA_HC_PERSIST_Q8=1` (with `STRATA_HC_PERSIST=1`) makes int8 copies of each hc read's weights at start-up, on Q8_0's grid: one fp16 scale per 32 values, the same rounding as `STRATA_HC_Q8`. The persistent kernel then reads those copies. The output changes, by about as much as `STRATA_HC_Q8` changes it.

Decode, 2x RX 6900 XT, 10 cold-start interleaved pairs, off vs on, same binary. The output text differs between off and on (int8 weights), so the draft acceptance on a given prompt differs too. **Time per verify window** is therefore the fair number:

| | off (median) | on (median) | paired difference (median) | pairs faster | range |
|---|---|---|---|---|---|
| 4K context, ms per window | 28.33 | 27.15 | **-4.2%** | 10/10 | -4.6 .. -4.0% |
| 32K context, ms per window | 27.11 | 26.54 | **-2.1%** | 10/10 | -2.3 .. -1.6% |
| prompt 32K, tok/s | 1,771 | 1,768 | -0.2% | | -0.4 .. +0.1% |

Plain decode tok/s on the same runs was +7.2% at 32K and -0.5% at 4K. Both differences mostly reflect acceptance: the windows went 126 -> 120 at 32K and 122 -> 128 at 4K, the same in every pair. I do not claim tok/s from two prompts. The prompt read does not use this path.

Quality (our production tree with the same kernel, teacher-forced KL against a double-precision reference, 8 conversation segments). Top-10 overlap with the reference:

| | top-10 overlap | vs. no int8 |
|---|---|---|
| neither | 91.36 | |
| `STRATA_HC_Q8=1` (upstream) | 90.96 | -0.4 pt |
| `STRATA_HC_PERSIST_Q8=1` (this PR) | 90.64 | -0.7 pt |

Mean KL was within the segment-to-segment noise for all arms (a single segment varies by up to 5x), so I do not read it either way. The start-up check reports a max relative difference of 0.00497 against the bf16 read. An earlier version of this patch used one fp32 scale per 128 columns, and one per row for `up`; it reported 0.00705 and lost 1.3 pt, which is why the grid is now Q8_0's.

## Related discussion

none (opt-in, decode verifier only; it follows the existing `STRATA_HC_Q8` / `STRATA_HC_PERSIST` pair)

## Approach

- `fused_gr_persist_q8_prepare` builds one read's copies at start-up, outside graph capture: `qd [NR][D]` (down rows, then inject rows) with `sdh [NR][D/32]` fp16 scale bits, and `qu [D][LR]` with `suh [D][LR/32]`. The copies are keyed by the read's bf16 `w_down`, so a read without copies keeps the bf16 kernel. `Verifier::init` calls it for each of this stage's layers and prints the count and MiB. Layers whose weights already carry `q8_scale` (`STRATA_HC_Q8`) are skipped.
- `q8g32_rows_kernel`: d = amax / 127 per 32 contiguous values, then q = round(x / d) with d already rounded to fp16, which is ggml's reference rounding.
- The persistent kernel gets a `Q8` template argument. In the down/inject phase each lane pair shares one 32-value group and its scale; in the up phase each thread applies five group scales.
- `fused_gr_persist_q8_on()` is true only after the int8 kernel passes a start-up check on the current card against the bf16 read (bar 2e-2; ~5e-3 measured).
- Cost: VRAM for the copies, 360 MiB on card 0 (54 reads) and 280 MiB on card 1 (42 reads) in a 27/21 layer split. On our two-card setup that took 196 expert slots off card 1 (-3.8%). On a single 16 GB card the copies (641 MiB) left the draft head without room in 1 of 2 cold starts, so a single 16 GB card needs a smaller `--vram-reserve` or context.

## Validation

Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X, 128 GB DDR4-3200, Ubuntu 26.04, kernel 7.0.0-42, ROCm 10.0 runtime (`/opt/rocm/lib`).

**Passed:**
- Build: main 54828c7e + this change, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_PREFILL_MMQ=ON -DSTRATA_Q6K_EXPERTS=ON -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DSTRATA_BUILD_TESTS=ON`.
- `ctest -j 1`: 221/224 (`hc_persist_parity` and `hc_persist_default_off` pass; they cover the bf16 read, not the int8 one). The three failures are the same on main 54828c7e without this change: `ple_parity` (Q2_0 gguf relative path), `conversation_copy_streams_test` (148 MiB first copy stream) and `isa_guard_check` (CPU iq512 function).
- Speed: the table above. IQ3_S pack, two-card layer split at 27, `--spec 3 --spec-min-p 0.6 --pipeline-windows 1 --kv int8`, parallel 1, `STRATA_HC_PERSIST=1` plus our usual HIP env. `benchmark.py` 4K + 32K, 256 tokens each, arms alternating. Each arm produced the same text in all 10 runs. 0 stalls.
- On our production tree (same kernel), the standard protocol's 12 sanity checks: 12/12, 0 stalls, with parallel 2 + `--batch-mtp`.

**Failed:**
- none observed

**Not run:**
- CUDA / SYCL / Metal. The kernel is in `fused_gr.cu` next to the existing persistent kernel. I built and ran only HIP gfx1030.
- A ctest for the int8 kernel. It is covered by the start-up check (int8 kernel on int8 copies of random weights against the bf16 read, bar 2e-2, 4.97e-3 measured), which turns the mode off on failure, but not by a standalone test. I can add one if you want it in ctest.
- Long-context headroom at 128K-256K with the copies' extra VRAM, on this PR's tree.
- Other gfx103x chips.

## Limits

- The gain is ~2-4% per window, smaller than halving the hc bytes would suggest. The hc read is one part of the window; experts and attention are the rest.
- The copies cost VRAM (see Approach). On 16 GB cards that is real. That is the main reason this stays opt-in.

## Update: `STRATA_HC_I8ONLY=1` - keep only the int8 copies

The Limits section above says the copies cost VRAM. This commit removes that cost. With `STRATA_HC_PERSIST_Q8=1` the decode never reads the bf16 hc weights any more; only the prompt read and the 1-token read do. `STRATA_HC_I8ONLY=1` (needs `STRATA_HC_PERSIST=1` and `STRATA_HC_PERSIST_Q8=1`, HIP) skips loading the bf16 hc projections and keeps only the int8 copies. The paths that take bf16 get it by expanding the int8 codes into one reused scratch per stream, the way `STRATA_HC_REQ8` already does for the prompt GEMM.

This is `STRATA_HC_REQ8`'s idea on a layer split (main still warns "STRATA_HC_REQ8 does not support a layer split yet"), with one difference: the copies are the persistent read's own (fp16 scale per 32 values, built by the same `q8g32_rows_kernel` from the same bf16), so the **decode kernel reads exactly the same int8 data as with `STRATA_HC_PERSIST_Q8=1` alone**. On our production tree a checksum of the copies (`STRATA_HC_Q8_HASH=1`) was the same with and without `I8ONLY` on both cards (not re-run on this tree).

The GGUF holds these tensors as BF16 (288 tensors, no Q8_0), so the copies are still built from the pack's bf16 at start-up: read once from the host into a temporary device buffer, quantized, freed (0.3 s per card).

**Measured on this PR's tree** (2x RX 6900 XT, layer split 27/21, IQ3_S, `parallel` 1, upstream-only switches; A = `PERSIST=1` + `PERSIST_Q8=1`, B = A + `I8ONLY=1`; 10 interleaved cold-start pairs, 20 starts, 0 stalls):

| | A | B | |
|---|---|---|---|
| bf16 hc loaded (card 0 / card 1) | 675 / 525 MiB | 0 | int8 copies 360 / 280 MiB in both |
| expert cache slots | 6,613 / 5,554 | **6,951 / 5,795** | **+338 / +241** |
| free with everything loaded | 466 / 424 MiB | 468 / 424 MiB | kept equal, see below |
| decode, 4K prompt | 83.4 tok/s | 85.8 | +2.9% (B ahead in 7/10 pairs) |
| decode, 16K prompt | 97.2 | 98.3 | +1.1% (5/10, noise) |
| prompt read 4K / 16K | 837 / 1,599 tok/s | 862 / 1,602 | +3.0% / flat |

To compare at the same headroom, B ran with `--vram-reserve-mib 252 --vram-reserve-later-mib 354` so it ends the start with the same free VRAM as A. With the default reserve, B's slots are only +100 / +73 and the rest stays free (914 / 770 MiB after load): with `PERSIST_Q8` alone the copies are made after the expert cache is sized and are paid from the 700 MiB reserve, with `I8ONLY` they exist before it is sized, so the same reserve leaves ~450 / 350 MiB unused. #5691 (measured reserve) would pick that up without a hand-set number.

So `PERSIST_Q8` + `I8ONLY` together use ~560 MiB *less* per pair of cards than neither, instead of 640 MiB more.

**Output:** the decode kernel is unchanged, but the prompt read now sees int8-rounded hc weights, so the KV after a prompt differs slightly and the text differs from `PERSIST_Q8` alone. On our production tree, teacher-forced KL against the double-precision reference, 8 segments, 4,067 positions: 0.0372 (PERSIST_Q8) vs 0.0329 (+ I8ONLY), argmax match 94.5 vs 94.7%, top-10 overlap 90.9 vs 90.7%. Single segments move by up to 3x between arms, so I read this as no change, not as better.

**Refusals:** `I8ONLY` refuses to start without `STRATA_HC_PERSIST=1` and `STRATA_HC_PERSIST_Q8=1`, with `STRATA_HC_REQ8` (two int8 layouts for one tensor), on non-HIP builds, and if the int8 kernel failed its start-up check (there would be no bf16 to fall back to). If a persistent read finds no copy for a weight that has no bf16, it aborts instead of reading garbage.

Validation of this commit on this PR's tree: B at 128K: 2,022 tok/s prompt, 80.7 tok/s decode, 0 stalls, lowest free 358 MiB (window graph capture). Standard protocol (B): 21/21, 12 of 12 sanity checks (instruction / retrieval / tool calls 4/4), 0 stalls. ctest `-j 1`: 221 of 224, the same three failures as this PR without the commit, none in a file it touches: `ple_parity` (its Q2_0 GGUF is not on this box), `conversation_copy_streams_test` (ROCm 10 takes 148 MiB for a first copy stream, the test allows 16), `isa_guard_check` (two AVX-512 functions in `iq_avx512.cpp` from main, `src/kernels/cpu` is untouched here). Not run: CUDA / SYCL / Metal (the bf16 expansion is in `fused_gr.cu`, HIP built only), single 16 GB card.

Ported from our production tree (where it has served since 2026-11-09 with slots +293 / +205 and time per verify window -3.2%); the port drops a DP-read hook that main does not have yet (#5002).


## Update 2026-11-15

Merged `main` (13eedf552) into the branch without conflicts and rebuilt (HIP gfx1030, ROCm 10.0). On the merged tree: ctest as before (the two failures are this machine's existing ones, `ple_parity` without the Q2_0 GGUF and `conversation_copy_streams_test` with ROCm 10's 148 MiB first copy stream), and the standard protocol with `STRATA_HC_PERSIST=1 STRATA_HC_PERSIST_Q8=1 STRATA_HC_I8ONLY=1` is complete, 21/21, 12 of 12 sanity checks, 0 stalls, with both cards printing the int8-only line (360 / 280 MiB). CI is green. Still not run: CUDA / SYCL / Metal on hardware (CI builds them; both modes are opt-in, so the default path is unchanged). Marking it ready for review.

## Update 2026-11-20: rebased onto v0.2.16 (main 58ee32277), plus a fix for a trimmed layer split

Rebased onto current `main` (`v0.2.16-4-g58ee32277`) as a linear branch: the earlier merge commits are gone, and the code moved to `engine/`. There is one new commit:

- **a5c0224b5**: `STRATA_HC_I8ONLY` on a trimmed layer split. Since #4522, CUDA0 reloads its dense weight table after the split is decided, to drop the layers it does not keep. CUDA0's int8 copies were made before that reload, so the reload dropped their marks, and CUDA0 lost its int8-only read. CUDA0's copies are now made after the trim, on the table it keeps. Without a trim, or on one card, nothing changes. After the fix both cards print the int8-only line again: CUDA0 54 reads of layers 0-26 (360 MiB, 675 MiB of bf16 not loaded), CUDA1 42 reads of layers 27-47 (280 MiB).

Re-measured on the rebased tree: same box, now 

Mehr auf der Site

Links zu Install, Modellen, Releases.