Pull requests / #2226
#2226 STRATA_HC_PERSIST=1 (opt-in, HIP): the verifier's hyper-connection read as one persistent kernel (gfx1030: read 1.35-2.10x, per window -4.2 % / -2.8 %)
open · @Vop3p · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDADocumentation
Beschreibung
## Title Issue: none of its own. `STRATA_GR_V3=1` is another opt-in for the same read; on gfx1030 it needs the two-half split, which fails its check (#375), so it falls back to the default read. ## Summary At 1-8 tokens the verifier's hyper-connection read (twice per layer) is three small launches with two global dependencies, and on gfx1030 the down projection runs 41 blocks on 40 WGPs. Opt-in `STRATA_HC_PERSIST=1` (HIP) runs it as **one kernel of 80 blocks with two grid barriers**: the down projection split by column slices over all 80 blocks, a distributed reduction, then the up projection and the same epilogue as `gr_up_kernel`. 2x RX 6900 XT (gfx1030), ROCm 10.0, layer split: - the read alone, 1-8 tokens: **1.35-2.10x** (38.9 us against 55.1 us at 2 tokens); - ms per verify window, this branch, one binary with the variable unset / set, 6 alternating rounds: **4K 25.81 -> 24.73 (-4.2 %), 32K 24.40 -> 23.72 (-2.8 %)**, every round the same way; tokens per window and prompt speed unchanged. Not bit-identical (another summation order). Against a double-accumulation reference (`STRATA_HC_PERSIST=3`) on the 106K-token chat of a 3-chat `STRATA_LOGPOS` run on this branch: KL 0.0131 / argmax 96.3 % against the default read's 0.0136 / 95.0 %. ## What changed - `src/kernels/cuda/fused_gr.cu`: the persistent kernel, its start-up check and `fused_gr_read_persist`, all inside `#if defined(__HIPCC__)`; CUDA and SYCL get two empty stubs. - `src/core/verify.cpp`: `Verifier::init` calls `fused_gr_persist_init()`; the multi read tries `fused_gr_read_persist` first and falls through to `fused_gr_read_multi` when it returns false (off, more tokens than were checked, a Q8 / QFUSE read). - `include/strata/kernels/fused_gr.hpp`: the two declarations. - `docs/AMD_HIP.md` (RDNA2 section): one bullet. - `bench/results/2026-10-12-hc-persist-gfx1030/`: the measurements, smaller cards simulated with `HSA_CU_MASK`, what was tried and dropped, and `hc_persist_bench.cpp` to time the read alone. How it stays safe on cards it was not written for: - wave64 is refused; - every token count is **run** once at start-up with a 50 ms barrier timeout and compared with the default read; a timeout keeps the default read for that count and larger ones (checked: 1 and 5 WGPs via `HSA_CU_MASK` time out and fall back, no hang). No occupancy API in front of it: it promised too many blocks on 30 WGPs and too few with Ubuntu's HIP 7.1 runtime, which turned the read off although it runs resident there; - one `strata hc:` line per card says what was decided. What it leaves alone: the default path (with the variable unset both entry points return in their first line), the drafter's reads (their own stream), the prompt path, CUDA, SYCL. Default path byte-identical: main (a0116b27) built without and with this change, same options, variable unset: the 106K chat's 481 scored positions are bitwise identical (the two shorter chats differ between two runs of the same binary on this machine, so they cannot show it). ## Extra Notes - Built: HIP (gfx1030) only. **CUDA and SYCL were not built** (no toolkit here); the new code is HIP-only and the other backends see two stubs and two declarations. - Only one card model was measured. Simulated 30 / 20 WGPs keep Navi 21's bus and Infinity Cache, so a real 6700 XT / 6600, RDNA 3, RDNA 4 are open. **Help wanted**: from any RDNA card, the `strata hc:` line and a `STRATA_DECODE_TIMING=1` run with and without `STRATA_HC_PERSIST=1`. - 0 stalls in every engine run with it (more than 30); in daily use on this machine since 2026-10-12. - Unrelated, seen once while checking: main's default path failed one 106K checkpoint save with "conversation snapshot running-state copy: an illegal memory access was encountered" (the variable unset, so this code did not run); a rerun passed. The same message as #1932; noted there would be the place if it comes back. Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030, PCIe 4.0 x8 each) / Ryzen 5 5600X, ROCm 10.0.
Mehr auf der Site
Links zu Install, Modellen, Releases.