Pull requests / #3052
#3052 docs(bench): dual PCIe V100 layer split with same-engine single-card control (0.1.44.1)
open · @noahark · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
Dual-GPU layer split on two PCIe V100-class cards, measured **against the same engine on one of the same cards** — same suite, same day, n=2 per arm. The community has dual-GPU reports on DDR4 hosts (#1300, #1028) but no same-machine single-card control, so the true split gain was not separable from the platform before; this fills that gap and adds a `--pipeline-windows 2` arm on NVIDIA (opt-in there), plus the first Windows + Volta (sm_70) dual-GPU datapoint we know of. ## Machine 2x Tesla PG500-216 32 GB (V100-class, 1,132 GB/s HBM2), PCIe 3.0 x16 each, TCC; dual Xeon E5-2696 v3, 128 GB DDR3L-1600; Windows 10; official prebuilt cuda12 engine **0.1.44.1**. ## Headline numbers (decode tok/s, 4.3K-doc leg) | | 1x | 2x split | 2x + `--pipeline-windows 2` | |---|---:|---:|---:| | IQ3_XXS (262K) | 77.0 | 108.6 (1.41x) | **129.0 (1.67x)** | | UD-IQ4_XS (131K) | 40.0 | **79.1 (1.98x)** | - | - Caches after the split (auto, K=24): **24,576 / 24,576 pairs (100%)** on XXS, **~99.8% of the routed mass** on UD — the DDR3L host stops mattering for decode almost entirely. - UD gains more than XXS: its 4-bit experts are the biggest per-token host traffic, so the single card was the most host-bound. - H1 probe (n! trailing zeros) correct in **17/17 runs** across all arms. - With the caches full: the `--pcie-frac` sweep is flat 0→1 (92.3-92.4 tok/s), and the single-card UD tier's 20% pool-workers sensitivity disappears. - `--pipeline-windows 2` on NVIDIA: +15-19% on top of the split; an H1 short-prompt read drops to 21 ms. Peak 62/60 C at the 250 W limits. ## Contents - `bench/results/2026-10-18-community-2x-pcie-v100-ddr3l/` — README with the full method, 12 raw result JSONs (both rounds of every arm), and `summarize-dual-0144.py` which regenerates the paired table from them. - `bench/results/COMMUNITY.md` — index row. - `docs/MULTI_GPU.md` — a Measured subsection with the paired table. Same platform family as our earlier single-card reports in #617 / #1355 (the DDR3L host whose decode sat at 33-42 tok/s single-card on the prebuilt-engine generations). Happy to run follow-ups (batching on a split, `STRATA_PREFILL_HELP`, a third card when it arrives) if useful.
Mehr auf der Site
Links zu Install, Modellen, Releases.