#2248 Benchmark on RTX 4060 (8 GB) + RTX 2060 (6 GB) — heterogeneous dual-GPU

本文

**Measured:** 2026-10-13  
**Result:** It helps — no regression. Decode holds ~36 tok/s on a 125B MoE.

This is a heterogeneous two-card setup: an Ada (`sm_89`) card paired with a Turing (`sm_75`) card, running Strata's pipeline layer-split. The 2060's 6 GB is below Strata's 8 GB recommendation, so the main question was whether it helps or drags. Answer: it helps — no regression, decode holds ~36 tok/s on a 125B MoE.

## Hardware and software

- **GPU:** NVIDIA RTX 4060 8 GB (Ada, `sm_89`) on PCIe 3.0 x8 + NVIDIA RTX 2060 6 GB (Turing, `sm_75`); used together via `--layer_split auto`, card order `1,0` (faster card last). Combined VRAM denominator observed: 14.3 GB.
- **CPU:** Intel Core i7-9700KF, 8 cores / 8 threads — AVX2 only, no AVX-512, no VNNI (CPU expert fallback runs on the slower AVX2 path).
- **RAM:** 62 GB installed.
- **Storage:** Kingston SA200 M.2 SATA 500 GB (SATA, not NVMe).
- **OS:** Ubuntu (exact version not recorded at measure time — 22.04 or 24.04).
- **Driver / CUDA:** nvidia-driver-595 (595.99.02), CUDA 13.0 / 13.2.
- **Strata:** commit `a0116b27` (tag `v0.1.43.1`), source build.
- **Engine:** compiled locally with `CMAKE_CUDA_ARCHITECTURES=75;89`, CUDA 13.0.88, host GCC 13.3.0, Ninja. NCCL not found (warning only — dual-GPU pipeline uses pinned-RAM hand-off, not NCCL).
- **Background:** desktop GUI (Xorg + gnome-shell + Firefox) active on GPU 0 during the run.

## Model and configuration

- **Model:** `Qwen/Qwen3.8-Flash-Next`, quantization `IQ2_XS` (GSQ-RCO), GGUF `Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001/00002-of-00002.gguf` (~68 GB total).
- **Vision encoder:** `mmproj-Qwen3.8-Flash-Next-BF16.gguf`.
- **MTP draft:** `q2_0`, ~0.889 GB GGUF → `Strata-data/mtp/rt` (29 tensors, speculative decoding).
- **Context:** 32768 tokens; KV int8; expert-cache auto; prefill auto.
- **Speculative decoding:** `--spec 4 --spec-min-p 0.5` (MTP on).
- **Vision on:** `--vram-reserve-mib 700`.

```text
# strata-iq2_xs.json (set by ./setup.sh --gpus 0,1 --model IQ2_XS --context 32768 --vision yes)
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 \
--max-context 32768 --kv int8 --vision --vram-reserve-mib 700 \
--server 127.0.0.1:8080 --gpu 0,1 --layer_split auto
```

```text
# engine log: [strata] layer split: card order 1,0 (the faster card last; "gpu_order": "as_given" keeps 0,1, #1352)
```

## Method

Used the built-in `./bench.sh` against the already-running server (no restart, settings above). Warm-up run not measured. Repetitions and caps as reported by `bench.sh`:

- **chat:** 3 short questions × 3 runs, up to 100 new tokens each.
- **4K prompt:** 3 fresh ~4096-token prompts, 24 new tokens.
- **16K prompt:** 3 fresh ~16384-token prompts, 24 new tokens.
- **agent:** a 6-turn session with 4 tools × 3 runs (prefix-cache reuse exercised).

`bench.sh` writes per-run data to:

```text
bench/results/2026-10-13-community-rtx-4060-plus-rtx-2060-qwen3-8-flash-next-iq2-xs/
```

`README + results.json + engine-log-excerpt.txt`; home/user/IP already anonymized.

## Results

### Token counts

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs |
|---|---:|---:|---:|---:|
| chat (3 questions) | ~53 | 0 | 100 | 9 |
| 4K prompt | 4019 | 0 | 24 | 3 |
| 16K prompt | 16058 | 0 | 24 | 3 |
| agent (6-turn, 4 tools) | 609 → 14468 | 0 → 11622 (68.7% hit) | varies | 15 turns |

### Speed

| Configuration | Prompt tok/s median (range) | Decode tok/s median (range) | TTFT seconds median (range) |
|---|---:|---:|---:|
| chat (3 questions) | — (short prompt) | 35.8 (34.1–43.8) | 0.93 (0.81–1.02) |
| 4K prompt | 287.7 (283.1–289.4) | n/r¹ | 14.01 (13.92–14.24) |
| 16K prompt | 298.5 (298.4–299.2) | n/r¹ | 53.84 (53.72–53.85) |
| agent (6-turn, 4 tools) | — | 36.1 (30.9–43.4) | 22.68 (11.22–25.78) |

¹ `bench.sh` reports prefill (prompt) tok/s for the long-prompt rows but not a separate decode figure for the 24-token generations; marked n/r (not reported).

### Extra observations

- VRAM hit rate on a cold start was ~38% (410 experts cached) and rose to ~1548 experts as the session warmed; the warm median decode (35.8) is the realistic steady-state number. An early single Monitor reading of 26.4 tok/s was a cold-cache artifact, not steady state.
- Combined VRAM 13.3 / 14.3 GB confirms both cards hold weights/expert cache.
- Agent prefix-cache hit 68.7% → only ~73 re-read tokens/session, efficient for tool loops.

## Correctness and limitations

- **Correctness:** model loads and answers coherently (spot-checked one Chinese prompt in the web chat; reply was on-topic). Full correctness suite not run.
- **2060 below 8 GB threshold:** Strata warns a sub-8 GB card "may run slower than without it." On this hardware it did not regress — decode stayed at ~36 tok/s — but the 2060's smaller expert cache means more CPU fallback than a ≥8 GB second card would give. Numbers are therefore a conservative lower bound for two ≥8 GB cards.
- **No AVX-512 on the CPU** caps the expert-fallback path; a CPU with AVX-512 would likely lift the decode median above 36.
- **NCCL absent:** only a build warning; pipeline layer-split via pinned RAM works without it.
- **Not recorded:** exact Ubuntu version, RAM single- vs dual-channel, sustained thermal after 10 min. Storage is SATA — irrelevant here since all experts live in RAM (only a small n-gram/lookup table touches disk).
- Compared to the prior FreeToken setup on the same box (`Qwen3.6-35B-A3B`, 26.3 tok/s), this runs a ~3.5× larger 125B MoE at 35.8 tok/s — the reason for switching.