#2038 [0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35%

Description

## Environment

- Engine: **v0.1.43 (c69adca)** — regression present. Good: **v0.1.41 (fb58e0d)**. Not bisected across 0.1.42.
- GPUs: 2× RTX 3090 48 GB (sm_86 build), driver 595.71.05, CUDA 13.0.88, Linux 5.15
- Model: Qwen3.8-Flash-Next UD-IQ4_XS (3-part GGUF, mmap)
- Serving flags (identical for both versions): `--pack --native --expert-profile --expert-cache auto --prefill auto --spec 5 --spec-min-p 0.5 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 262144 --batch-mtp --resident-budget-gib 60`, layer split auto (2 stages: CUDA0 dense+early layers, CUDA1 rest+expert arena), parallel 4, port 6006

## Repro

Serve with the flags above on a 2-GPU layer split, then send one 200k-token prompt, 1024 output tokens, greedy-ish (reasoning off). Compare decode tok/s between v0.1.41 and v0.1.43. Reproduces every run; variance between runs is the tell (below).

## Measured impact (same machine, same config, same bench)

| Metric | 0.1.41 | 0.1.43 | Δ |
|---|---|---|---|
| solo 200k decode | 131.4 / 130.8 tok/s (runs: 135.8, 131.4, 130.8 — tight) | 39.0 / 57.0 / 59.9 / 76.2 tok/s (2× run-to-run variance) | **−42…−70%, unstable** |
| prefill (200k) | 3934 / 3914 tok/s | 2540–2670 tok/s | **−35%** |
| TTFT cold | 50.5 s | 74–78 s (when prefill regressed) | — |
| ×4 aggregate (batch, cache-aligned) | 126.5–129.8 | 96.3 | −25% |

## Evidence

1. **Decode window launch serialization.** `STRATA_DECODE_TIMING=1`: per-window `stage` time 0.64 ms (0.1.41) → **29.57–40.60 ms** (0.1.43), while the GPU kernels themselves only grew 16.6 → 23 ms. Consistent with the host synchronizing per layer instead of enqueueing the two-stage pipeline ahead — the two-GPU overlap is gone.
2. **dmon during decode**: 0.1.41 is a flat 348 W / SM 92% on both cards. 0.1.43 shows the two cards **alternating idle** (SM 0–34%), power oscillating 200–335 W — classic ping-pong of a broken pipeline.
3. **Page cache thrash**: per-request major faults 0 (0.1.41) → **5139–8941** (0.1.43); server log shows lines of the form "OS read 500–1700 MB for 0.0 MB expert reads" — mmap'd GGUF pages being re-read from the OS while the engine believes it issued no expert reads.
4. Prefill and decode regress by the same magnitude and share the instability → likely one common root cause in the shared launch/scheduling path, not two separate issues.

## Ruled out

- `STRATA_MMVQ_IL=0` (kernels 195d379) — no change (also: we're IQ4_XS, not Q3_K)
- Revert of 6b39264 (verify service doorbell) — no change
- cgroup memory pressure — 73/180 GB used, no reclaim pressure
- Expert residency — `--no-prefill-borrow` (100% VRAM-resident experts) still reproduces
- Profiling overhead — ~10% on solo only, batch unaffected, can't explain 2×

## Suspects from the 0.1.41..0.1.43 range (analysis, not verified)

- **aa7b0cb** "CUDA: the default --pcie-frac is refined in the first decode windows from the CPU pool's measured speed (**single GPU**, no --pcie-frac)" — if the refinement engages on a layer split (where "CPU pool speed" probing may misbehave across stages), it would plausibly stall windows and cause the mmap re-reads in (3). Untested opt-out on our side: `STRATA_PCIE_FRAC_DEFAULT=old`. **Happy to test this first if you think it's plausible.**
- **5ef500d / b97a36a** prefill step records from a pinned buffer — candidate for the prefill half, but wouldn't explain decode.
- The prefill lend mechanism was independently excluded via `--no-prefill-borrow`.

## Side note: STRATA_SPLIT_MTP_BATCH works

On 0.1.43, `STRATA_SPLIT_MTP_BATCH=1` + `--batch-mtp` on a layer split does function (4 slot drafters on CUDA1, 83.4% acceptance, 7.25 rows of the 8-row window at ×4), but net aggregate was 96.3 vs 129.8 on 0.1.41 — i.e., the feature's gain is currently masked by this regression. Worth re-benchmarking once the regression is fixed.