Pull requests / #3232

#3232 [Benchmark] 3x RTX 2080 Ti 22 GB (sm_75): v0.1.45 vs v0.2.0-beta, UD-Q4_K_XL at 262K, layer split 16,32

open · @Adamyno · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

Beschreibung

## Problem and result

`docs/TEST_REQUESTS.md` asks for a report from "Three or four NVIDIA GPUs", and there is no sm_75 multi-GPU report in `bench/results`. This adds one: three RTX 2080 Ti 22 GB in a layer split on a 125B MoE at its native 262,144-token window, measured on 0.1.45 (2026-10-18) and re-measured on **0.2.0-beta** (2026-10-19) on the same rig and config.

Result on 0.2.0-beta: 65.2 tok/s short-chat decode, 586.8 / 1,345.2 / 1,758.7 tok/s prompt reading at 4K / 16K / 32K, 69.7 tok/s agent decode at a 69.3% conversation-cache hit. The 0.1.45 → 0.2.0-beta change is inside this box's run-to-run spread (+1.7%, +0.9%, +0.2%, −0.1%, −2.8%).

## Related discussion

`docs/TEST_REQUESTS.md`, row "Three or four NVIDIA GPUs". No issue tracks this report.

This supersedes #2976, which was closed in the 0.2.0-beta tracker reset. GitHub would not reopen it after the rebase (the branch had been force-pushed), so the same branch is opened again with the 0.2.0-beta re-measurement added.

## Approach

Two report folders, same hardware, same engine arguments, so they compare directly:

- `bench/results/2026-10-18-community-3x-2080ti/` — Strata 0.1.45
- `bench/results/2026-10-19-community-3x-2080ti-0.2.0-beta/` — Strata 0.2.0-beta (engine 0.1.45.3, `3d1fb480`), with the 0.1.45 comparison table in its README

`results.json` records a layer-split run as one card and leaves `quant` / `config` empty (#2887); those fields were filled in by hand from the server's start line and `nvidia-smi`, and the README says so. The measured numbers are untouched.

## Validation

**Passed:** `python tools/strata_bench.py --url http://127.0.0.1:8097 --long` against a server already running, three runs per prompt size, 263 s for the suite, `results.json` → `failures` empty. Build: source at the `v0.2.0-beta` tag (`3d1fb480`) with `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75`; `cuobjdump --list-elf` lists sm_75 cubins only.

**Failed:** none observed.

**Not run:** correctness, needle or token-identity checks (this is a speed suite); an IQ-family pack on three cards (Q4_K_XL uses the K-quant expert rows, so the engine's AVX-2 path did the expert work); full-power cards (the power limits are 210 / 167 / 200 W).

## Limits

Speed only. The absolute numbers are below what the same rig does at the default 250 W per card. One model, one quantization, one context length (262,144 tokens). The agent rows are 15 requests, so their medians are the weakest in the report.

Mehr auf der Site

Links zu Install, Modellen, Releases.