Pull requests / #2379

#2379 bench: community reports, 1x/2x/6x Tesla V100-PCIE-32GB scaling, Unsl…

open · @zwcf5200 · 0 comentarios · En GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsSecurityDocumentationLinux

Descripción

Results-only PR: four report folders in bench/results/ (README + per-run results.json each), four
COMMUNITY.md rows and one docs/COMMUNITY_BENCHMARKS.md entry. No engine changes.

Hardware: 1x / 2x / 6x Tesla V100-PCIE-32GB (sm_70, PCIe, auto layer split on the multi-card arms),
2x Xeon Silver 4116, 502.6 GiB RAM, Linux, driver 580.178.04, CUDA 13.0. Engine 0.1.43.1 (a0116b2)
as installed, Unsloth UD-Q4_K_XL with its expert profile, --spec 4 --mtp rt --kv int8
--resident-budget-gib 71.

Results (tools/strata_bench.py --long, 3 runs per cell, medians):
- Fresh-prompt prefill at 4K/16K: 596 / 1,017 (one card), 992 / 1,646 (two), 1,281 / 2,422 tok/s
  (six, 32K context). Short-chat decode: 37.5 / 63.3 / 70.6 tok/s across the ladder.
- The same six cards at 262,144 context take a 32K prompt at 3,388 tok/s and decode 80.1 tok/s.
- Agent sessions reuse 68.5-68.8% of the follow-up turns' tokens (5-turn sessions, ~14.5K tokens).

To my knowledge this is the first six-V100 measurement on record (previous largest: 4x, #2048).
Complements #1508 (2x PCIe V100, IQ4_XS, older engine) and issue #2260 (4x PCIe 32GB, same engine).

Limits: speed only (no needle or correctness runs), one machine, power limits and background load
not recorded, 32K context skipped on the 32,768-token arms.

Prepared with an AI assistant (Zed agent) from the runs on the machine above; the numbers are the
author's own measurements.

En el sitio

Enlaces a install, modelos, releases.