Pull requests / #2032

#2032 community bench: 2x RTX 2080 Ti (sm_75, NVLink unused), IQ3_S, 0.1.43

open · @arbulin · 0 comments · View on GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quants

Description

community bench: 2x RTX 2080 Ti (sm_75, NVLink unused), IQ3_S, 0.1.43

Speed-only report from `tools/strata_bench.py --long` (3 runs per test,
medians with ranges), measured on the machine's own HTTP API.

- Hardware: 2x RTX 2080 Ti 22 GB (44 GiB VRAM), i5-11500T 6C/12T,
  62.6 GiB DDR4-2933, Ubuntu (kernel 7.0.0-28), driver 610.57.04
- Engine: 0.1.43 source build (commit 7695859a), sm_75
- Model: original Flash-Next IQ3_S at the native 262144-token context,
  layer split 24 across the two cards, `--kv int8`, `--prefill auto`
- Results: chat decode 78.9 tok/s; fresh prompts 873.7 / 1521.9 / 1776.9
  tok/s at 4K / 16K / 32K; agent session 68.8% cache hit

The 32K prompt number is about 15% above the 2x TITAN RTX (also sm_75)
report at 0.1.40.3 on the same model and context.

Two notes from this machine that may interest others:

- `STRATA_BF16_TC=1` and `STRATA_PREFILL_HELP=1` were both set. The first
  is a no-op here (7.x cards take the FP16 tensor-core path by default);
  the second measured +18% on short prompts in our own A/B.
- The conversation cache budget only matters if the admission rule can be
  met: with a full expert arena in RAM, MemAvailable after load was about
  2.58 GB, below the "snapshot + 2560 MiB floor" every time, so a large
  `--conversation-cache-mib` produced 166 rejected parks, 21 evictions and
  a first-token spike to 72 s. Lowering the budget to 2048 MiB removed
  both the rejections and the spike and left prompt throughput unchanged.

No correctness check was run; background load and power limits were not
recorded (the cards were on their default policy during the runs).

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.