Pull requests / #2976

#2976 bench: community report, 3x RTX 2080 Ti 22 GB (sm_75), v0.1.45, Q4_K_XL at 262K

open · @Adamyno · 0 comments · View on GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsDocumentationLinux

Description

Adds `bench/results/2026-10-18-community-3x-2080ti/` from a 3-card Turing rig, the hardware `docs/TEST_REQUESTS.md` asks for ("Three or four NVIDIA GPUs") and a real sm_75 card for the Turing series.

**Hardware:** 3× RTX 2080 Ti 22 GB (sm_75, power limits 210/167/200 W), i9-7980XE, 125.5 GiB DDR4-3600, Linux, CUDA 13.0, driver 580.178.04, layer split 16,32.

**Build/config:** v0.1.45 (`477ac13c`) source, `CMAKE_CUDA_ARCHITECTURES=75`; Flash-Next UD-Q4_K_XL, 262,144 ctx, `--kv int8 --spec 3 --spec-min-p 0.54 --batch 2`, `STRATA_SM75_PERF` unset (release default).

**Results (`tools/strata_bench.py --long`, 3 runs, medians):**
- short chat decode **64.1 tok/s** (58.4–81.2), TTFT 0.53 s
- 4K / 16K / 32K fresh prompts: **581 / 1,343 / 1,761 tok/s** prompt, TTFT 6.9 / 12.0 / 18.3 s
- agent session: **71.7 tok/s** decode, 68.8% conversation-cache hit

The folder also carries three addenda measured on the same rig: a **one-hour soak** (505 requests, 0 errors, no drift, max 49 °C), `STRATA_SM75_PERF=0` vs `1` (2 cards: +1.5% decode at 10.5K/25K, +0.8–1.5% prefill), and **0.1.43.1 vs 0.1.45** (3 cards, default-vs-default: 10.5K decode −2.3%, prefill +0.1%).

**Tool note:** `strata_bench.py` recorded the layer-split run as one card (#2887); `results.json` was corrected to the real three-card set, power limits, quant and config, and the README says so.

Not measured: correctness/needle checks, full-power cards (the limits are tuned for tokens-per-watt).

Related on this site

Quick links to related pages.