#6156 [Engine] Qwen3.6-35B-A3B on 2x RTX 3060: qwen35moe ignores multi-GPU flags — patch adds a card-1 expert tier, 37.6 → 74.8 tok/s (+97%)
open · @Mark-1982 · 4 comments · View on GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentationLinux
Description
ystem: Linux (Proxmox LXC), 2x RTX 3060 12 GB (24 GB total, sm_86), Intel Xeon W-2133, 72 GB RAM
Strata 0.2.15 / Terminal 0.1.61 / Engine 0.1.60 (cuda13 component)
Model: Qwen3.6-35B-A3B Q8_0 (37.8 GB) from the pinned Unsloth feed
What happens
--------------
The qwen35moe engine path is single-GPU. Every multi-GPU mechanism the other families use is
rejected at start:
strata launch --gpu 0,1 -> "strata-q36: ignoring --layer-split auto (not an option of this engine)"
strata launch --gpu-roles 1=main,0=helper -> same; nvidia-smi after start: GPU0 11,367 MiB, GPU1 283 MiB
--set peer-device=1 -> "strata-q36: ignoring --peer-device 1 (not an option of this engine)"
Placement at 32k context, Q8_0, one card:
strata-q36: hybrid experts: 7 of 256 per layer in VRAM (0.89 GiB), all 32.67 GiB in pinned host
memory, the rest of a batch of <= 128 rows on 6 CPU threads
Context is capped by the single card, not by RAM (72 GB free):
32768 -> loads, 26.3 tok/s decode (measured, /v1/status predicted_per_second)
65536 -> cudaMalloc(&E.stage[b], max_layer + kMmqTail) failed: out of memory (q36_engine.cu:3142)
also fails with --kv int8 and --kv q4_0
262144 with --set kv-resident=20480 -> "a context of 262144 tokens does not fit on this graphics card:
its conversation memory needs 11.56 GiB and only 2.19 GiB is free after the model.
Start it again with a shorter context (48128 tokens fit here)"
Expected / why this matters
---------------------------
24 GB of VRAM exist but only 12 GB are usable, so the KV for 64k+ cannot be placed even though
32.67 GiB of experts already sit in pinned host memory. A second card as either a layer-split stage
or an expert-cache tier (docs/SECOND_GPU.md, --expert-cache-device1) would give both capacity and
context. llama.cpp on the same two cards already runs the same GGUF at 131072 ctx
(--n-cpu-moe 32, q8_0 KV) at 15.1 tok/s, so the ceiling here is the engine path, not the hardware.
Strata's single-card 26.3 tok/s is why we would rather not use llama.cpp.Related on this site
Quick links to related pages.