#2662 bench: community report for 2x RTX 2080 Ti on a dual-socket Xeon (split, concurrency, vLLM)

本文

## Summary

Adds a community benchmark report for **2x RTX 2080 Ti 22 GB over a dual-socket Xeon
E5-2696 v4** (44 physical cores, AVX2 only, 94 GiB RAM, NVLink), indexed in
`docs/COMMUNITY_BENCHMARKS.md`. No code changes.

The report combines the repo's own `tools/strata_bench.py` output (`results.json`,
every request) with a hand-written A/B analysis, and fills three gaps the earlier
2x-2080 Ti report (`2026-10-06-community-r9-2x-rtx-2080-ti`) left open:

1. a **single-GPU vs layer-split** comparison on the same cards,
2. **concurrent-request throughput** with `--batch`,
3. a **same-workload vLLM comparison** on the same hardware.

## Canonical suite (`tools/strata_bench.py --long --clients 4 --runs 3`)

Final config: layer split + `--batch 4 --batch-groups 2`. Medians with ranges.

| Configuration | Prompt tok/s | Decode tok/s | TTFT s |
| --- | ---: | ---: | ---: |
| short chat | — | 42.8 (31.9–49.7) | 0.67 |
| 4K / 16K / 32K prompt, fresh | 678.6 / 1102.4 / 1298.1 | — | 5.96 / 14.62 / 24.76 |
| agent session, follow-up turns | — | 50.3 (39.0–65.5) | 5.26 |
| 4 clients, 128 new tokens each | — | 41.3 total | 2.68 p50 |

## Single GPU vs layer split

| metric | single | split |
| --- | ---: | ---: |
| decode (424 / 2,944 / 11,582-token prompts) | 28.4 / 29.8 / 31.3 | **52.4 / 57.9 / 57.3** |
| window | 70.06 ms | **37.96 ms** |
| CPU pool | 9.52 ms | **0.79 ms** |

The decode gain is almost entirely the CPU pool collapsing — on a CPU this slow the second
card removes expert work, it does not just add memory.

## Concurrency + `--batch`, and the power limit

`--batch 4 --batch-groups 2` packs several requests into one verify window: concurrency 4
goes 62.2 → **75.1** tok/s, concurrency 8 → **82.9** (warm-cache decode). Raising the power
limit 180 → 250 W adds +21% (75.1 → 90.8 at concurrency 4). Quality 31/31 on the 7-category
suite.

## vLLM on the same cards

| concurrency | Strata + batch (250 W) | vLLM (250 W) | ratio |
| ---: | ---: | ---: | ---: |
| 4 | 90.8 | 37.1 | 2.45x |
| 8 | 92.0 | 37.0 | 2.49x |

vLLM stays at ~37 tok/s whether the limit is 180 W or 250 W (FP8 is emulated on SM 75);
Strata fills the GPU with concurrency + `--batch` and only then becomes power-bound.

## Notes / not tested

- The engine actually run is **0.1.43.1** (the fork's project metadata says 0.1.44.1); the
  report states this. Building 0.1.44.1 was out of scope for this report.
- Thermal soak, the 128K needle cases at 250 W, and a quant-matched vLLM run (the vLLM side
  runs FP8) were not done. The `--batch` and 250 W tables are single batches, not the
  10-round protocol some other reports use. All stated in the report's Limitations.