#2939 4× Tesla T10 (Turing, layer split): single-stream decode stays ~39 tok/s with 99.9% of experts in VRAM — same as 2× T10

Descrição

**Hardware**
| | |
|---|---|
| Motherboard | Huananzhi H12D-8D (single socket SP3), BIOS 2.2 (2026-01-06), no BMC |
| CPU | AMD EPYC 7452, 32 cores / 64 threads (Zen 2), 1 socket, 1 NUMA node |
| RAM | 4 × 32 GB Samsung DDR4-2400 RDIMM (M386A4G40EM2-CRC) = 128 GB, 4 of 8 memory channels populated |
| GPUs | 4 × Tesla T10 16 GB (TU102, sm_75, VBIOS 90.02.41.00.01), passive cooling, power limit 100 W each (default 150 W) |
| PCIe | every card on its own CPU root port, Gen3 x16 (LnkCap 8 GT/s x16); `nvidia-smi topo -m`: NODE between all pairs; P2P works with `iommu=pt` (10–13 GB/s per pair) |
| Storage | Crucial T500 1 TB NVMe (model files on it) |

**Software**
- Ubuntu 24.04.3, kernel 7.0.0-38-generic, kernel parameter `iommu=pt`
- NVIDIA driver 595.91.07 (server, open kernel modules), CUDA 13.2 toolkit
- Strata v0.1.45 (477ac13c), engine compiled locally with CUDA 13.2 (no ready-made engine for v0.1.45: releases/latest returned 404)
- Qwen3.8-Flash-Next GSQ-RCO IQ3_S, `--spec 4 --spec-min-p 0.5 --mtp rt --kv int8 --kv-resident 32768 --max-context 262144`, `"gpu": [0,1,2,3]`, `"layer_split": "auto"`

**What the log says**
- `layer split auto: K=14,26,38 - predicted 44.8 ms per decode window; the caches hold 21931 of 24576 profiled pairs (~99.9% of the routed mass)`
- `CUDAn 56 SMs at 1.40 GHz -> 0.93 ms per layer`, every link probed at 13.1 GB/s (pcie_frac 0.36)
- `adaptive overlap: off: not on a layer split`
- After load: CUDA3 has 341 MiB VRAM free and gets the "LOW ... add --vram-reserve-later-mib" warning
- MTP draft acceptance on 300-character Chinese answers: roughly 95–119 of 166–203 drafts (~55–65%)

**Measured** (6 Chinese prompts, ~300 characters each, no thinking, temperature 0, warm-up + median of 3 rounds)

| | tok/s |
|---|---|
| single stream decode | 39.3 |
| same, with `"parallel": 4` | 38.9 |
| 2 concurrent (`parallel` 4) | 36.5 / 38.9 per stream |
| 4 concurrent (`parallel` 4) | 27–32 per stream, 104 total |
| prompt read, 2.6K / 10K tokens | 527 / 723 |

For comparison, #2007 reports 44.2 tok/s on **2×** T10 with v0.1.40.1 and 40.5 on 0.1.43 (the regression between 0.1.40.1 and 0.1.40.2 noted there). Our 4-card result is below both, even though four cards hold almost every expert.

**Questions**
1. With nearly all experts resident, single-stream decode looks bound by the four stages running one after another (~0.93 ms per layer × 48 layers ≈ the predicted 44.8 ms window). Is there a way to overlap the stages for one stream on 3+ NVIDIA cards (`--pipeline-windows` mentions two GPUs / AMD gfx12 only), or is 4-way split simply not expected to beat 2-way for a single user?
2. Is the 0.1.40.1 → 0.1.40.2 slowdown from #2007 still present in 0.1.45 on Turing? We can run an A/B (0.1.40.1 vs 0.1.45) on this box if useful.
3. Would `--prefill-pipe` or a manual `layer_split` be worth trying for prompt reading here (723 tok/s at 10K tokens)?

Happy to run any flag combination or a debug build — the box is free for testing.