#3497 [Engine] Turing (sm_75), 2× Tesla T10 layer split: both GPUs idle 17 % of decode; 6 % is the host-side cudaGraphLaunch of the ~1,050-kernel verify graphs (1.5 ms each)

本文

**What happened**

An nsys profile of single-stream decode on two Tesla T10s (sm_75, layer split 24/24) shows that both GPUs are idle 17 % of the time. Only about a third of that is waiting for expert rows over PCIe. The largest share that no GPU work covers is the host launching the verify-window graphs: each one holds about 1,050 kernel nodes, and `cudaGraphLaunch` takes a median of **1.47 ms** host time per launch (p90 1.53 ms). No kernel runs on either card during that time.

Strata's own overlap options (`STRATA_SPLIT_OVERLAP=1`, `--pipeline-windows 2`, also with `STRATA_MTP_STAGE0=1`) are all slower on this box (numbers below). So the launch cost is not hidden by them on Turing.

**Strata version**

v0.2.1.3 (cbfd5b92, engine 0.1.46.2), source build with `CMAKE_CUDA_ARCHITECTURES=75`, nothing patched. The same profile on v0.1.45 (477ac13) gives the same picture within 0.5 points on every line below.

**Graphics card(s)**

2× Tesla T10 16 GB (TU102, sm_75), power limit 100 W each, PCIe 3.0 x8/x8 (the link probe reads 6.7 GB/s on both). Driver 580.178.04, CUDA 12.9.

**Memory and system**

AMD Ryzen 7 5700G (8C/16T), 64 GB DDR4, ASUS PRIME X570-PRO, Ubuntu 24.04, kernel 7.0.0-38.

**Model and config**

Qwen3.8-Flash-Next GSQ-RCO IQ3_S native pack, `--layer-split 24 --expert-cache auto --prefill auto --spec 4 --mtp rt --spec-min-p 0.70 --max-context 245760 --kv int8 --kv-resident 32768 --pcie-frac 0.00 --prefill-pipe 1536`.

Env: `STRATA_SM75_PERF=1 STRATA_EXP_MODE=8 STRATA_QSA_ROWS=1 STRATA_ATTN_DAILY=1 STRATA_SCORES_THREAD=1 STRATA_KV_PREFETCH=1 STRATA_CACHE_MESSAGE_BOUNDARY=1 STRATA_SIDE_MIXER=2 STRATA_B6_MMVQ_ROWS=1 STRATA_SELECT_SIMT=1`. This is the fastest set on this box (#2197).

**How it was measured**

```
nsys profile -t cuda,nvtx --cuda-graph-trace=node --trace-fork-before-exec=true --delay=150 --duration=20 \
  python serve/server.py --engine strata --config <config> --port 8091
```

The capture is taken after load and warm-up. It covers one greedy request with thinking off and 3,000 max tokens: a Python module plus a test suite. The request generated 2,216 tokens at 74.4 tok/s (the server's own timing, with nsys attached). The analysis is the union of kernel intervals (and copy intervals) across both devices, from the exported SQLite file.

**Where the time goes (19.5 s window, v0.2.1.3)**

| | share of the window |
|---|---|
| GPU0 (layers 0-23) busy | 35.5 % (spin-waits `wait_flag_ge` / `wait_copy_i32`: 2.4 %) |
| GPU1 (layers 24-47 + head + draft) busy | 47.2 % (spin-waits: 2.1 %) |
| **neither GPU runs a kernel** | **17.3 %** |
| of that: H2D copies running (expert misses, 18.3 GB at 0.94 GB/s) | 6.4 % |
| of that: gaps under 10 µs between kernels (678k gaps) | 3.8 % |
| of that: gaps of 1 ms or more (813 gaps) | **6.2 %** |

**The 1 ms gaps are the graph launches**

| graph | launches | `cudaGraphLaunch` host time, median |
|---|---|---|
| verify windows, 1,034-1,086 kernel nodes | 891 | **1,465 µs** (p90 1,533 µs) |
| 53-65 nodes | 1,223 | 114 µs (p90 196 µs) |
| 4-5 nodes | 814 | 24 µs (p90 32 µs) |

In the gaps of 1 ms or more, the only host call in flight is `cudaGraphLaunch`. That is about 1.4 µs per node, so the cost scales with the node count. On this driver/arch the launch is not cheap enough to hide behind the previous window.

Top kernels by GPU time, for context: `native_mmvq_il_kernel` 19.8 %, `native_gu_fused_kernel` 13.3 %, `native_down_lds_kernel` 8.0 %, `gr_down_staged_kernel` 7.8 %, `gr_up75_kernel` 7.2 %, `native_q6_k_mmvq_kernel` 5.6 % (GPU1 only, the head). No single kernel dominates.

**What I tried: Strata's overlap options on the same box**

Method: `tools/ab_engine.py`, fresh server per arm, 3 rounds with alternating order, greedy, 512 new tokens, three chat prompts plus a 21,102-token prompt. Paired median decode against the config above. These A/Bs were run on 0.1.45; 0.2.1.3 ran at the same speed as 0.1.45 in a separate A/B (−0.9 % decode, within noise).

| arm | decode | prefill (21K) |
|---|---|---|
| `STRATA_SPLIT_OVERLAP=1` (log: "one hand-off per window (overlapped)") | −1.7 % | −1.1 % |
| `--pipeline-windows 2` | −3.1 % | −0.9 % |
| `--pipeline-windows 2` + `STRATA_MTP_STAGE0=1` | −4.7 % | −5.3 % |
| `--layer-split 26` / `27` instead of 24 | −1.2 % / −1.5 % | −2.3 % / −4.6 % |

With `--pipeline-windows 2`, short prompt reads got faster (chat prompts 78 → 105 tok/s), but decode lost on every request type.

**Question**

On sm_75, could the verify window be launched with fewer graph nodes, for example by fusing more of the small per-layer kernels or by grouping per-layer work into fewer launches? Or could the next window's graph launch be started earlier on the host so that its 1.5 ms overlaps the current window?

The 1 ms gaps plus the sub-10 µs gaps are about 10 % of decode on this box. Even part of that would be the largest remaining sm_75 gain I can find without new kernels.

I can run any branch, switch or debug build on these cards and send nsys exports (the `.nsys-rep` files are 20-25 MB).

**Reproduction status:** measured locally, every number here is from my runs on this box.