Pull requests / #2628
#2628 gfx12 layer split: opt-in STRATA_GFX12_SPLIT_DECODE=1 for nine decode switches (2x R9700: +5.8% greedy, +4.9% sampled)
open · @justincinmd · 0 comentários · No GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descrição
Issue: follows #1478 ## Summary #1478 left the decode half of the gfx1151 table out of the gfx12 table, because it measured slower on one R9700. On a two-R9700 layer split, with every expert resident, the same switches are faster. This PR adds an opt-in flag for that case: `STRATA_GFX12_SPLIT_DECODE=1`. The default does not change. Measured on 2x R9700 32 GB, Linux, ROCm 7.2.4, Qwen3.8-Flash-Next IQ3_S, `--layer-split 26`, all 24,576 experts resident, spec 4 + MTP: | | Greedy tok/s | Temp 1.0 tok/s | |---|---:|---:| | v0.1.43, before | 108.9 | 101.5 | | v0.1.43 + the nine switches | 115.0 | 106.2 | | v0.1.43, after | 108.5 | 101.0 | That is +5.8% greedy and +4.9% sampled against the mean of the two stock runs. Decode is the geometric mean of four prompts, 3 runs each. Cold prompt reads at 4.6K / 19K / 69K tokens are +11% / +12% / +12% (one request per size). Per-run data and output hashes: [bench report](https://github.com/justincinmd/Strata/tree/pr/bench-2x-r9700/bench/results/2026-10-12-community-2x-r9700-iq3_s-0.1.43) (also offered as a PR). ## What changed - `STRATA_GFX12_SPLIT_DECODE=1` (exactly `1`) sets nine switches of the gfx1151 table, unless the user set them: `Q6_PACKED`, `MMVF_ROWS`, `ATTN_LANECELL`, `EXPERT_V2`, `TSUM`, `LFUSE`, `GDN_SPLIT`, `QFUSE`, `PLE_BATCH`. - They apply only when the resolved split runs on two or more GPUs, every one gfx12, on Linux. Otherwise one start-up line says why they stay off. The engine decides after the split is resolved and before any of them is read. - Without the flag, a qualifying split prints one hint line. `STRATA_GFX12_DEFAULTS=0` still turns off the whole gfx12 table. - `src/core/arch_defaults.cpp` / `.hpp`: a pure decision function, `gfx12_split_decode()`. `arch_default_env` and `apply_arch_defaults` keep their 0.1.43 signatures and output. - `src/program/generate.cpp`: one call, HIP only, after the existing check that every used GPU has code in the binary. - docs/AMD_HIP.md: what each switch does, the measurements and the limits. Short notes in DETAILS.md and MULTI_GPU.md. Not in the set: `Q8_PACKED` (the measured pack has no Q8_0 matrix it packs) and `SH_STREAM` (not measured on a split). `Q6_PACKED` costs VRAM: +497 MiB for the main head and +81 MiB for the MTP head on this pack. ## Testing - Without the flag the output equals stock v0.1.43 on every greedy and seeded request, and the start-up log adds only the hint line. With the flag the nine switches turn on, and expert residency does not change. With `--split-device 0` or one visible GPU they stay off. - Output with the flag: greedy reasoning text, answer text and token counts match stock on 4 prompts in 3 runs. 11 of 12 seeded sampled runs match. Stock v0.1.43 itself varies on some seeded requests from run to run, so this does not show whether the switches are bitwise on gfx12. The new start-up line calls them speed switches, not exact. - `arch_defaults_test` covers the default, the flag, one GPU, mixed architectures, Windows, malformed values and user-set values. - CUDA 13.0.2 (sm_120) and oneAPI DPC++ 2026.1.1 (SYCL) build with no new diagnostics. Compile only. Windows not built. ## Extra Notes The evidence covers one setup: 2x R9700, one IQ3_S pack, every expert resident. If more gfx12 split runs agree, the set could become a default later.
No site
Links install, modelos, releases.