#2086 Community benchmark: RTX 3090 + RTX 5070 + RTX 5070 Ti layer split (Q2_0)

本文

Results-only submission. No engine changes.

**Hardware:** RTX 3090 (24 GB) + RTX 5070 (12 GB) + RTX 5070 Ti (16 GB), i7-8700K, 64 GB RAM, model shards on a spinning HDD. No 32 GB card in the machine.

**Model:** ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, Q2_0, custom pack via tools/iq_pack.py. Engine 0.1.42 local source build.

**Config:** gpu [0,1,2], layer_split 28,40, --pipeline-windows 2, --trim-stage-weights, --kv int8, --kv-resident 16384, --max-context 260000, STRATA_SPLIT_OWN=1, STRATA_PREFILL_HELP=1. Engine reports 99% of profiled expert pairs resident.

**Headline numbers** (3 runs each, greedy, 256-token cap, salted prompts so nothing was prefix-cached):

| Config | Prompt tok/s median (range) | Decode tok/s median (range) | TTFT median (range) |
| --- | ---: | ---: | ---: |
| short (756 tok) | 1,175 (1,058-1,176) | 129.2 (117.1-133.8) | 0.66 s (0.66-0.73) |
| long (44,736 tok) | 3,275 (3,274-3,276) | 109.2 (108.0-114.0) | 13.71 s (13.70-13.72) |

Needle recall: 6/6 at 32k and 128k, depths 10/50/90.

**Main limitation:** the 12 GB RTX 5070 stage holds 5,922 of its 6,144 profiled expert pairs; the 222 non-resident pairs stream from RAM on every prompt chunk. Split position also matters a lot - the 5070 Ti has the slowest PCIe link of the three (3.0 GB/s engine probe), and moving layers onto it degrades prefill sharply.

Two values are marked `not measured` in the README: the PCIe link width (lspci -vv returns no LnkCap/LnkSta on this host) and the Hugging Face revision of the Q2_0 shards (not recorded at download time; byte sizes are given instead).

Files: README.md, runs.json (per-run engine timings + TTFT + total latency + VRAM/RAM snapshots), needles.json, community_bench.py.