#3161 bench: community report, 2x RTX 2080 Ti 22 GB (sm_75), IQ3_S — v0.1.43→v0.1.45 decode A/B + SM75_PERF 90K prefill +13.7%

Description

# Field report: 2x RTX 2080 Ti 22 GB (sm_75), Flash-Next IQ3_S — v0.1.43 → v0.1.45 controlled A/B, and the sm_75 switches on a deep-context chat workload

Speed-only, two controlled A/Bs on one live production box:

1. **Engine 0.1.43 vs 0.1.45** on the same config (`--spec 4`, binary swap, same day): prefill **+4.9 %**, decode **-4.5 %** across three fixed tasks, end-to-end **-5.8 %**. The decode direction is consistent across 4 rounds x 3 tasks; sample sizes are small (see caveats).
2. **`STRATA_SELECT_SIMT=1` + `STRATA_SM75_PERF=1` on/off**: prefill unchanged at 21 K, **+13.7 % at 90 K** cold reads. Kept on. Consistent with #3056's depth-growing curve; our gain is larger, plausibly because IQ3_S routes more experts.

## Hardware and software

- **GPUs:** 2x RTX 2080 Ti 22 GB (modded VRAM), sm_75, passed through to an LXC container
- **Host:** Proxmox 6.17.13-1-pve, Intel Xeon E5-2696 v3 (36 threads), 62 GB RAM; container CT103
- **Driver:** 610.57.04. Engine binaries: official 0.1.45 tar (`BUILD.json: archs [75], cuda-13.1p`) and the kept `strata.bak-v0143` from the previous install
- **Model:** Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S, native IQ pack, MTP draft layer (`--mtp rt`), `--spec 4 --spec-min-p 0.5 --kv k8v4 --max-context 198000 --layer-split auto --expert-cache auto --prefill auto --vision --vram-reserve-mib 700 --remote-expert-opt --control-vector-scaled ...:1.0 --cvec-mode project --kv-grow`
- **Env (baseline, already set before this test):** `STRATA_BF16_TC=1 STRATA_IO_PREFETCH=1 STRATA_ADAPT_OVERLAP=1 STRATA_STAGE_TRIM=1 STRATA_DISJOINT_ADAPT=1`

## Method

- **Decode:** fixed 3-task prompt set (code generation 500 tok / long explanation 700 tok / copy-edit of 600 config lines 600 tok, `temperature: 0`) against `/v1/chat/completions`, warmup first. Client e2e tok/s + server-log decode tok/s and draft acceptance. 0.1.43: 2 rounds; 0.1.45: 4 rounds.
- **Prefill:** dedicated probe — unique random-word prompts (~21 K and ~90 K tokens, fresh seed per run so no KV reuse), `max_tokens: 1`, server-log read tok/s.
- Engine swapped by file replacement + service restart; config byte-identical between arms except the two env keys in part 2.

## Part 1 — 0.1.43 vs 0.1.45 (same config, `--spec 4`)

Decode, server-side tok/s (per-task means over rounds):

| task | 0.1.43 (n=2) | 0.1.45 (n=4) | delta |
| --- | ---: | ---: | ---: |
| code_gen | 75.4 | 74.5 | -1.2 % |
| long_explain | 77.7 | 73.1 | -5.9 % |
| repeat_edit | 93.7 | 88.1 | -6.0 % |
| **mean** | **82.3** | **78.6** | **-4.5 %** |

Draft acceptance (round-by-round):

| task | 0.1.43 | 0.1.45 |
| --- | --- | --- |
| code_gen | 67.4 / 72.7 % | 70.8 / 62.1 / 64.5 / 76.8 % |
| long_explain | 79.4 / 69.3 % | 69.8 / 70.4 / 75.9 / 67.2 % |
| repeat_edit | 96.8 / 91.2 % | 81.3 / 90.4 / 84.8 / 82.7 % |

Prefill (21 K cold, server read tok/s): 1584.7 (n=2) -> 1663.0 (n=4), **+4.9 %**, every 0.1.45 round above every 0.1.43 round. Client e2e mean: 76.0 -> 71.6 (**-5.8 %**).

Per-round 0.1.45 decode (code_gen / long / repeat): 77.0/74.6/87.7, 67.5/73.0/89.4, 69.7/75.7/86.9, 84.0/69.2/88.3. Round-to-round spread is +-10 %; the version direction is consistent across all rounds and tasks.

Two notes that may matter to you:

- The kept baseline binary reports `strata 0.1.43` — i.e. **pre-hotfix**. Since 0.1.43.1's notes say Linux NVIDIA desktop decode was regressed in 0.1.43 ("the adaptive overlap is opt-in again", "two-card batch serving ~35 % faster again"), 0.1.45 losing 4.5 % decode to a *known-regressed* build looks worse than the raw number.
- The release notes' speed claims for 0.1.44/0.1.45 are measured on Qwen3.6 / R9700 / AMD / Arc; the notes also state Flash-Next answers are byte-identical to 0.1.44.1, so no decode-path change for this model was intended. Our numbers suggest something in the decode path did move for sm_75 + IQ3_S. The prefill +4.9 % is consistent with the bigger default prompt chunks.

## Part 2 — sm_75 switches, depth-dependent (this box)

`STRATA_BF16_TC=1` was already on. Adding `STRATA_SELECT_SIMT=1` + `STRATA_SM75_PERF=1`:

| cold prompt | off (server read tok/s) | on | delta |
| ---: | ---: | ---: | ---: |
| ~21 K | 1605.3 / 1685.7 / 1646.2 / 1714.9 (mean 1663.0) | 1583.0 / 1739.6 (mean 1661.3) | 0 % |
| ~90 K | 2031.5 / 1974.5 (mean 2003.0) | 2282.7 / 2271.3 (mean 2277.0) | **+13.7 %** |

Decode unaffected within noise. Kept on permanently. The depth-growing shape matches #3056 (+2 % at 2 K -> +6.7 % at 77 K); our 90 K gain is larger — this box runs IQ3_S with `--kv k8v4` and a control vector, so more routed expert work per token.

## Caveats

- n=2 (0.1.43) vs n=4 (0.1.45); round-to-round variance +-10 %; directions consistent, exact magnitudes not trustworthy.
- The box serves a live chat agent during the runs (visible as 40-90 K prompts interleaved in the log); both arms carry the same interference.
- No correctness suite was run; the sm_75 bitwise claim is taken from #3056, not re-verified here.

## Reproduction

Scripts (fixed-prompt decode suite `bench_spec_ab.py`, cold-prefill probe `prefill_probe.py`, aggregator) and raw JSONL + per-request server logs are in this repo snapshot: branch `local/ct103-20261019`, paths `bench/spec-ab-20261019/` and `bench/version-ab-20261019/` (self-hosted Gitea: http://192.168.2.11:3200/hermes/strata-ct103). Happy to upload them here as gists/attachments on request.