Pull requests / #3411
#3411 perf(cuda): enable Volta defaults with dual V100 benchmark evidence
open · @koshikawa-masato · 0 comments · View on GitHub
Description
## Change V100 (sm70) currently receives no table from `nv_default_env`, even though manually enabling a subset of the newer CUDA optimizations improved decode on my dual V100 PCIe 32 GB system. Add that six-switch table for sm70, retain explicit user overrides, and provide `STRATA_SM70_PERF=0` to opt out. Existing homogeneous-device gating remains unchanged. The table enables SIDE_MIXER=2, GR_UP75=1, B6_MMVQ_ROWS=1, GU_SWIGLU=1, QSA_ROWS=1 and ATTN_DAILY=1 (all `STRATA_`-prefixed). It does not change sampling, VRAM reserves or CPU-expert settings. ## Measurements Ryzen 9 9950X / 128 GB RAM / 2 × V100 PCIe 32 GB at 150 W each, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, MTP4, 32K context, INT8 KV. Official v0.2.0-beta engine; same binary/config/model in all arms. Three repeats per workload, baseline → stack → baseline, then stack after long-context stress. | Arm | Japanese decode | Code decode | |---|---:|---:| | Baseline before | 74.23 tok/s | 108.40 tok/s | | Six-switch stack | 78.63 tok/s | 116.77 tok/s | | Baseline after | 74.40 tok/s | 108.70 tok/s | | Stack after stress | 78.70 tok/s | 116.87 tok/s | Approximately **+5.8% Japanese / +7.6% code** versus pooled baseline. All nine output texts matched in every arm (hashes included). Two concurrent 32,000-token inputs + 512 generated tokens, repeated twice, passed both sentinel checks without GPU errors. Long-context pair latency remained ~44.3 seconds; no improvement claimed there. Included report, per-request measurements/output hashes and exact short-prompt measurement script under `bench/results/2026-10-20-v100-sm70-defaults/`. ## Validation and limits - New CPU-only `nv_defaults_test` passes on macOS and Linux with C++17, `-Wall -Wextra -Werror`: architecture dispatch, opt-out, explicit user overrides, idempotency, and existing sm60/sm86 dispatch. Registered with CTest. - `git diff --check` passes. - Source patch is against current main; GPU measurements were on the stated release, not a full GPU rebuild of this main commit. - Combined-stack experiment only, no per-switch ablation. Decoded-text equality is not a token-ID/logit/exact_pp identity gate. - Single V100, 16 GB cards, SXM2 and other models remain unmeasured. Additional hardware validation before broad rollout would be welcome. - Both arms used later-GPU reserve 650 MiB to retain all 12,288 expert pairs / zero-doorbell graph. Minimum GPU1 free memory was 267 MiB. This is deliberately not proposed as a global memory default. These are community diagnostic measurements, not the standard benchmark protocol. No private prompts or credentials are included.
Related on this site
Quick links to related pages.