#6529 [Engine] Volta sm_70 decode optimizations, zero-VRAM in-place Q8_0, NVLink split head, and volta-forever detection
open · @midhatn · 0 コメント · GitHub で見る
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
本文
## Problem and result
**Problem:** Volta (`sm_70`) had no dedicated architecture default table in `engine/src/core/nv_defaults.cpp` (only `8.6`, `12.0`, `6.0`, and `12.1` existed), lacked tuned Volta `native_mmvq_il` row tables, decoupled `b6_launch`, and coalesced 4-block `Q8_0` hyper-connection (`GR`) kernels, paid ~248.7 MiB on GPU 0 and single-GPU LM head latency during `--peer-device` runs, could not run split-plane `Q8_0` decode GEMV without duplicating weight VRAM (`STRATA_Q8_PACKED`), and mis-identified GPUs as Turing `7.5` (and hit CUDA 13 cuBLAS heap exhaustion when `CUDA_MODULE_LOADING` stayed eager) under `volta-forever`.
**Result:** Measured on dual Tesla V100-SXM2-16GB (`sm_70`, NVLink `NV6`) locked at `200 W / 1020 MHz SM / 877 MHz HBM2` (`ecc=0`, zero throttle events across all samples), running `Qwen3.8-Flash-Next` (`512x56B`, `UD-IQ2_XS-v31-Q8`, `--kv int8 --peer-device 1 --peer-reserve-mib 1600 --draft dflash2 --spec 6`, `STRATA_ROUTE_TAIL_SKIP=0`, `STRATA_PF_PEER_STREAM=0`, `STRATA_P2P_DECODE_SCATTER=1`) across 2 interleaved rounds (`A->B->C->D` and `D->C->B->A`) x 3 seeded prompts (`64K` 3-needle retrieval `256` tokens, C++20 lock-free ring buffer `384` tokens, technical prose `384` tokens = `6` decode samples per primary arm; `1` round `64K` for `STRATA_PF_WMMA=1`):
| Arm | Configuration | GPU 0 / 1 slots | 64K prefill (tok/s) | 64K TTFT (s) | Decode median, n=6 (tok/s) | 64K / Code / Prose decode (tok/s) | Window median (ms/w) | Parity & 64K needles |
| --- | --- | ---: | ---: | ---: | ---: | --- | ---: | --- |
| `base` | `origin/main` (`STRATA_PF_WMMA=0`) | 5357 / 8587 | 1,450.0 | 42.64 | 68.05 | 68.05 / 74.0 / 63.8 | 41.53 | baseline (`3/3` needles) |
| `stageb_default` | zero manual Stage B flags (19-switch `sm_70` default) | 5357 / 8587 | 1,524.8 (+5.16%) | 40.56 | 84.60 (+24.32%) | 84.6 / 91.8 / 78.3 | 33.46 (-8.07) | bit-exact SHA on all 3 prompts (`3/3`) |
| `stageb_optin` | `+ STRATA_PEER_HEAD_SPLIT=1 + STRATA_Q8_INPLACE=1` | 5508 / 8430 | 1,522.7 (+5.01%) | 40.61 | 90.65 (+33.21%) | 90.65 / 99.3 / 85.05 | 33.05 (-8.48) | opt-in FP sum order + `+151` GPU0 slots (`3/3`) |
| `stageb_serving` | `+ draft_vocab_en.bin --draft-conf 0.45` | 5589 / 8425 | 1,522.7 (+5.01%) | 40.61 | 92.75 (+36.30%) | 92.75 / 97.85 / 87.0 | 31.39 (-10.15) | self-deterministic across rounds (`3/3`) |
| `stageb_pf_wmma` | `stageb_serving` + `STRATA_PF_WMMA=1` (`n=1`, 64K only) | 5589 / 8425 | 1,644.1 (+13.39%) | 37.62 | 89.50 (+31.52%) | 89.5 / — / — | 37.65 | FP16 TC MoE prefill (`3/3`) |
## Related discussion
- Credits and references:
- **[nbald/volta-forever](https://github.com/nbald/volta-forever)** — runtime `volta_forever_version()` / `volta_forever_abi()` detection (`crates/vf-proxy/src/marker.rs`) and `CUDA_MODULE_LOADING=LAZY` preservation guidance (`docs/guides/strata.md` section 5) for running CUDA 13 builds on Volta (`sm_70`).
- **[sybil-solutions/inference-optimisation-skill](https://github.com/sybil-solutions/inference-optimisation-skill)** — stage-level bottleneck accounting, lossless draft vocabulary pruning (`draft_vocab_en.bin`), `T=1..8` direct unit parity harnesses, and interleaved multi-prompt qualification methodology.
- **[jesdga95/basalt#442](https://github.com/jesdga95/basalt/pull/442)** — peer-GPU LM head split architecture (`NativeHead::run_verify_split` design inspiration).
- Based cleanly on `origin/main` (`6e38f588a`) and independent of companion PRs #6023 and #6210.
## Approach
1. **`volta-forever` detection & `sm_70` compute-capability mapping (`emulate.hpp`, `nv_defaults.cpp`, `qsa_prompt_attn.cu`, `generate.cpp`, CMake `dl` linkage):** Dynamically probes `volta_forever_version` in `libcuda.so.1` via `volta_forever_active()`, maps emulated `(7, 5)` back to Volta `(7, 0)` through `cc_of(major, minor)` across all architecture gates and kernel selectors, preserves `CUDA_MODULE_LOADING=LAZY` when active, and fixes the `#if defined(STRATA_EXPERIMENTAL_SM60)` guard in `qsa_prompt_attn.cu` so CUDA 13 builds still pick up the native Volta tensor-core prompt attention kernel.
2. **Bit-exact 19-switch `sm_70 (Volta)` default table (`nv_defaults.hpp`, `nv_defaults.cpp`, `nv_defaults_test.cpp`):** Enables 19 bit-exact switches automatically on uniform-`sm_70` / single-V100 runs (`STRATA_ADAPT_OVERLAP=1`, `SM70_TABLE=1`, `B6_MMVQ_ROWS=1`, `TSUM=1`, `SMALL_UNR=3`, `GR_NORM_FAST=1`, `SIDE_MIXER=2`, `ARGMAX_ROWS_SM70=1`, `LFUSE=1`, `GDN_SPLIT=1`, `SH_STREAM=1`, `Q80_ROWS1=1`, `NO_MMVQ_ROWS2=1`, `CPU_IQ_ALL_EXACT1=1`, `GR_Q8_DOWN_MODE=2`, `GR_Q8_UP_COLS=16`, `GR_Q8_UP_MODE=2`, `SCORES_HALVE=1`, `SCORES_THREAD=1`), with full rollback via `STRATA_SM70_PERF=0` or `STRATA_DEFAULTS=026` and explicit user-env precedence.
3. **Volta decode kernels (`native_mmvq.cu`, `fused_gr.cu`):** Adds Volta `mmvq_il` row thresholds and decoupled 6-token `b6_launch`, plus coalesced 4-block `Q8_0` hyper-connection down/mid/up kernels (`STRATA_GR_Q8_DOWN_MODE=2`, `STRATA_GR_Q8_UP_COLS=16`, `STRATA_GR_Q8_UP_MODE=2`) verified bitwise identical (including `q8_mixed`) across `T=1..8`.
4. **Opt-in zero-VRAM in-place `Q8_0` (`STRATA_Q8_INPLACE=1`, `STRATA_Q8_INPLACE_MIN_OUT`):** Repacks eligible dense `Q8_0` matrices in-place at host load time into split int8/fp16 planes (`+0.0 MiB` VRAM, using `q8_sm60.cuh` with Volta `6` blocks/SM default) plus bit-exact prefill dequantization (`dequant_q8_0_packed_kernel` in `dequant_bf16.cu`). Kept opt-in because FP accumulation order differs (`~1.1e-7` to `1.3e-7` relative RMS).
5. **Opt-in 50/50 P2P LM head split (`STRATA_PEER_HEAD_SPLIT=1`, `native_head.cpp`, `verify.cpp`, `verify_kernels.cu`, `dflash2.cpp`) + Metal stubs, register & docs:** Splits the `Q6_K` output head evenly across GPU 0 and `--peer-device` using CUDA-graph-safe `cudaMemcpyDefault` transfers and device argmax half-merging, freeing ~248.7 MiB on GPU 0 (`+151` resident expert slots); updates `engine/metal/src/engine/stubs_0145.cpp`, `terminal/data/settings-register.json`, `docs/NVIDIA_V100.md`, `docs/DETAILS.md`, and `docs/SETTINGS.md`.
## Validation
**Passed:**
- Windows 11 / MSVC v18.10.4 (`vcvars64.bat`, engine code tested on revision `d35cbf092`; final PR commit `7f2174765` only appends the B9 table to `docs/NVIDIA_V100.md`): `python tools/check --only cmake-cpu` -> `136 passed, 0 failed, 3 skipped` (`nv_defaults_test`, `adapt_fetch_default_test`, `arch_defaults_test`, and the full CPU suite passed).
- Docs & settings register on final commit `7f2174765`: `python tools/settings_register.py --check` (`exit 0`), `python tools/check --only docs` (`6 passed, 0 failed`), and `git diff --check` (clean, no output).
- Linux CUDA 12.9 (`sm_70`, dual Tesla V100-SXM2-16GB NVLink, `CMAKE_CUDA_ARCHITECTURES=70`, `STRATA_EXPERIMENTAL_SM60=ON`, revision `d35cbf092`): `cmake -B build -DSTRATA_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=70 && cmake --build build --target strata -j"$(nproc)"` succeeded; unit/parity binaries passed: `nv_defaults_test: ok`, `adapt_fetch_default_test: ok`, `arch_defaults_test: OK`, `mmvq_il_parity: OK (every case, T 1-8, rows 1/2/4 and the table, bitwise)`.
- Linux CUDA 12.9 HC parity harness (`test_gr_v2_parity`, `T=1..8`, modes `0..2` x `qf={0,1}` checking `mix`, `R`, `inj`, `rs`, and `q8_mixed` QFUSE outputs, excluding intermediate `lo` scratch): `total_bytes=11377152, byte_mismatches=0`.
- Linux CUDA 12.9 in-place `Q8_0` parity harness (`test_q8_inplace_parity`, Volta `BPS=6` default across all 6 dense matrix shapes `10240x2560`, `6144x2560`, `2560x6144`, `12288x2560`, `2560x640`, `640x2560`): `full_mm=0 ld_slice_mm=0 unaligned_mm=0` on all 6 shapes; `MMVQ` `T=1..8` `worst_rel_rms=1.270e-07`, `PARITY SUMMARY: failures=0`.
- Linux zero-flag smoke (`bench_v0217.py --name b5_zero_flag_smoke --bin ./build/strata --ctx 64k --mode peer1 --minimal-env`, locked `200 W / 1020 MHz SM / 877 MHz MEM`): verified `strata: sm_70 (Volta): 19 exact speed switches on by default` banner, `prefill=1517.3 tok/s`, `decode=83.4 tok/s`, `codes=True(3/3)`, exact baseline SHA `affdf99307c6`, `ecc_max=0`.
- Linux multi-prompt interleaved qualification (`run_b9_qual.py`, locked `200 W / 1020 MHz SM / 877 MHz MEM`, 2 rounds x 3 prompts x 4 primary arms + 1 `STRATA_PF_WMMA=1` arm): `hw_lock_and_ecc_ok=true`, `hw_violations=[]`, `all_needle64k_codes_ok=true`, `stageb_default_sm70_banner_ok=true`, and `stageb_default` matched `base` SHA bit-for-bit on all 3 prompts across both rounds (`affdf99307c6`, `ab03a61bae67`, `28bd3e2b6a1e`).
**Failed:**
- None observed.
**Not run:**
- HIP / ROCm, SYCL, and Metal GPU compilation and runtime tests (no AMD, Intel, or Apple GPU hardware available; Metal link stubs added in `engine/metal/src/engine/stubs_0145.cpp` and CPU/docs checks passed).
- Live CUDA 13 execution under `nbald/volta-forever` (test workstation runs CUDA 12.9; `cc_of()` remapping and `volta_forever_active()` gating were exercised via unit tests in `nv_defaults_test` and `adapt_fetch_default_test`).
## Limits
- `STRATA_Q8_INPLACE=1` alters FP32 dot accumulation order (`~1.1e-7` to `1.3e-7` relative RMS), so it remains opt-in. When combined with `STRATA_PEER_HEAD_SPLIT=1` (`+151` GPU 0 expert slots) in `stageb_optin` (`106K` draft vocab), rounds 1 and 2 followed two greedy trajectories (`35365e4689` vs `77c813528c` on `needle64k`, matching through the first 280+ characters and passing `3/3` needles in both rounds) due to near-tie FP rounding interaction with dynamic expert cache swaps (`stageb_default` matched `base` bit-for-bit on all 3 prompts across both rounds, and `stageb_serving` was self-deterministic across both rounds).
- `STRATA_PEER_HEAD_SPLIT=1` is opt-in because it requires `--peer-device` with ~249 MiB+ free peer VRAM and fast P2P bandwidth (NVLink).
- Benchmarks were collected on one dual V100-SXM2-16GB NVLink system locked at `200 W / 1020 MHz SM / 877 MHz HBM2`; other V100 systems with different clock/power caps or PCIe topologies may scale differently.
関連リンク
インストール・モデル・リリースへのサイト内リンク。