#2281 [0.1.43.1] 2× RTX 3090 CUDA layer split: decode −20.5% vs v0.1.43, and STRATA_ADAPT_OVERLAP=1 cannot re-enable the overlap ('off: not on a layer split')

Descrição

## Environment

- **Bad:** v0.1.43.1 (`a0116b2`, built from the `v0.1.43.1` tag tarball, `project(strata VERSION 0.1.43.1)`, self-reports `strata 0.1.43.1`)
- **Good:** v0.1.43 (`7695859`, the exact binary our production runs)
- GPUs: **2× RTX 3090 48 GB** (sm_86), driver 610.57.04, CUDA 13.4.92, Debian 13 container (LXC), PCIe 3.0 x16, measured link **7.9 GB/s**
- CPU: Xeon, 24 physical cores, `cpuset 0-23,48-71`, `memlock unlimited`
- Model: Qwen3.8-Flash-Next **UD-Q4_K_XL** (4-part GGUF, 103.69 GiB), native pack, PLE table on the 2nd shard
- Layer split auto (2 stages: CUDA0 then CUDA1)

Serving flags (**identical for every arm**, this is our production config verbatim):

```
--pack <pack> --native <shard1.gguf> --expert-profile <profile.bin>
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <rt>
--max-context 262144 --kv k8v4 --kv-resident 32768 --ple-io ram
--gpu 0,1 --layer-split auto
```

## Summary

**On this machine v0.1.43.1 is ~20.5% slower than v0.1.43 on decode and ~10% slower on a long prompt, and no environment variable recovers it.** This is the *opposite* direction from #2038 / #1964 / #1975 / #2010, where 0.1.43's adaptive overlap was the problem and 0.1.43.1 was the fix. For us the overlap (default on in 0.1.43) is a **+28% win**, and 0.1.43.1 loses that win while `STRATA_ADAPT_OVERLAP=1` no longer turns it back on.

## Measured (official `tools/ab_engine.py`, interleaved arms, order reversed each round)

Each arm n=8 decode samples (4 requests × 2 rounds). Request set: 3 short chats + one 6,345-token long prompt, greedy, `--max-new 256`.

| Arm | Binary / env | decode median | long-prompt prefill | Δ vs 0.1.43 |
|---|---|---:|---:|---:|
| **V43** | v0.1.43 (overlap default **on**) | **82.2–82.7 tok/s** | **1074 tok/s** | baseline |
| **V431** | v0.1.43.1 (overlap default **off**) | **64.7 tok/s** | 74.7 | **−21.7%** |
| **OVL1** | v0.1.43.1 + `STRATA_ADAPT_OVERLAP=1` | **64.4 tok/s** | 75.7 | **−22.1%** |
| **OVLH** | v0.1.43.1 + `OVERLAP=1` + `STRATA_ADAPT_SIGNAL=host` | **65.3 tok/s** | 73.3 | **−20.5%** |
| **OVLHW** | OVLH + `STRATA_ADAPT_CALLBACK_WARMUP=1` | **65.4 tok/s** | 73.1 | **−20.5%** |

Per-request medians (arm 4 run, n=8 per cell):

| request | V43 decode / prefill | OVLH decode / prefill |
|---|---|---|
| chat0 | 83.7 / 91.0 | 69.1 / 71.4 |
| chat1 | 81.9 / 82.8 | 58.5 / 70.2 |
| chat2 | 77.5 / 59.9 | 61.2 / 76.9 |
| long (6,345 tok) | 82.7 / **1074.4** | 70.8 / **717.9** |

**The regression is repeatable and order-independent**: round 0 vs round 1 (arms reversed) — V43 82.6 → 82.7, V431 64.3 → 66.5, OVL1 64.2 → 64.6. Not thermal/state drift.

Baseline sanity: the V43 binary is the one our production serves with, and production measures 72–86 tok/s on the same flags — so 82 tok/s is the true 0.1.43 number here, not a lucky run.

## Key evidence: the overlap cannot be re-enabled in 0.1.43.1

Start-up log with `STRATA_ADAPT_OVERLAP=1` explicitly set:

```
strata serve: STRATA_ADAPT_OVERLAP=1 is off: not on a layer split
```

…even though the same run logs `layer split across 2 GPUs: CUDA0, then CUDA1`. So the overlap's code path in 0.1.43.1 is not reachable in this configuration, and `STRATA_ADAPT_SIGNAL=host` does **not** restore the 0.1.43 speed (65.3 vs 64.7 without it).

This suggests **the 20.5% is not the overlap itself** but something else in the 0.1.43.1 range that is on our hot path. Our leading suspect from the commit list is `7bab660b` *"Prompt path: every pageable async host-to-device copy goes up through pinned bounce slots (#1996)"* — at 7.9 GB/s PCIe with a 2-stage layer split, that is exactly the kind of change that could cost ~20%. **We have not bisected this**, so treat it as a hypothesis, not a finding.

## Ruled out

- Not the adaptive overlap being *enabled* (it is off in every 0.1.43.1 arm, and forcing it on changes nothing)
- Not `STRATA_ADAPT_SIGNAL` (host callback restored: no change)
- Not `STRATA_ADAPT_CALLBACK_WARMUP` (no change)
- Not the config: both binaries run the **same 24-argument config file**, same `cwd`, same `serve/server.py`, only `exe=` differs
- Not `--batch` / `--batch-mtp`: we use neither (single request, no `parallel`)
- Not #2038's cause: our run has no `--batch-mtp`, and the log shows `--batch-groups` is not involved
- Not PLE I/O: `--ple-io ram`, table locked in RAM in 2.5 s in both versions

## Impact / decision

We are a production deployment on this exact hardware. **We are staying on v0.1.43** and skipping 0.1.43.1 — giving up the fixes we would otherwise want (#2026 `</tool_call>` handling, #2040 PLE auto-mmap, the serve-side BGEN/crash-recovery fixes) because a 20.5% decode regression is not an acceptable trade.

## Ask

1. Does the log line `STRATA_ADAPT_OVERLAP=1 is off: not on a layer split` on a 2-stage CUDA layer split look correct to you? If the overlap is meant to be usable on a split, this may be a second bug on top of the regression.
2. Is there a knob to test the #1996 pinned-bounce-slot path, or a commit to bisect between `7695859` (v0.1.43) and the 0.1.43.1 tag? We can run any interleaved A/B you want — we have the harness, the machine, and reproducible order-independent numbers (a full 3-arm × 2-round sweep takes ~26 minutes of exclusive GPU time).

## Raw data

Full JSONL (one row per request: round, arm, decode, prompt, prompt_n, completion n) available on request, as are the engine start-up logs for each arm.