#5857 [Benchmark] 2× RTX 3060 12 GB, vfio-passthrough KVM VM — qwen3.8-flash-next iq3_s, layer split

closed · @phamid · 1 commentaires · Sur GitHub

BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsSecurityDocumentation

Description

**What did you measure, or what went wrong**

Measured the same Qwen3.8-Flash-Next lane on one vs two consumer GPUs inside a **PCI-passthrough KVM VM** — a configuration the docs don't cover yet ("several NVIDIA cards in one PC" is true for virtual ones too).

**Hardware:** HP Z840 (dual Xeon, 56 threads), TrueNAS 25.10 host running a Ubuntu 24.04 KVM/OVMF guest, 56 vCPU, 96 GB RAM, no balloon. GPUs: 2× GeForce RTX 3060 12 GB (one LHR `2504`, one non-LHR `2503`), each passed through whole (video+audio) via vfio-pci; no NVLink or peer access; each card on its own CPU root, Gen3 x16 links, driver 580.119.02 in the guest.

**Strata:** 0.1.43.1, model `qwen3.8-flash-next-turbo-brilliance-abliterated-iq3_s`, context 262 144, MTP drafter active, OpenAI-compatible API on :8080.

**Configuration:** run-config `"gpu": 0` (baseline) → `"gpu": [0, 1]` with `--layer-split auto` (engine log: `layer split across GPUs [0, 1] (auto)`).

**Numbers:**
- Single card (baseline, steady): **38 t/s** decode.
- Dual, trial 1 (expert caches cold→warming): **43.6 t/s** (600-token completion, temp 0.7).
- Dual, trial 2 (warmer): **56.0 t/s**. Steady fully-warm figure not yet re-measured — that is the honest caveat: +15 %…+47 % so far, warm number pending.
- During generation both cards were genuinely busy: 48 %/57 % util, 11.4 GB/11.5 GB resident (each card caches experts for its own layer range, as the docs describe).

**Command:** OpenAI-compatible `POST /v1/chat/completions`, `max_tokens=600, temperature=0.7`, timed client-side; correctness spot-check returned the right 600-token content (and `17*23 → 391` in a separate probe).

**Poor results / gotchas worth recording:**
- First dual-GPU start took **~6 minutes** to `/health` 200 (both cards' expert caches load cold from NVMe; a single-card start was ~2 min).
- Layer split at 262 144 ctx leaves only ~0.5 GB VRAM headroom per card — fine so far, but KV growth beyond long contexts may need `--max-context` trimmed.
- Host-side (not Strata's fault, but virtualization users will hit it): OVMF wedged executing the GA106 VBIOS ROMs until the VM was started with `-global vfio-pci.rombar=0`; the guest nvidia driver loads its own GSP firmware afterwards and works normally. Also TrueNAS's boot-time vfio bind list silently ignored the second card until its initramfs script was patched.

**Reproduction status:** Measured locally

Co-authored-by: Hermes Agent (ff-z4) <[email protected]>

Sur le site

Liens install, modèles, releases.