#2138 [Bug]: v0.1.43 STRATA_ADAPT_OVERLAP + --batch 2: ~21% throughput regression (2x RTX 3090 / dual EPYC, controlled 2x2)

Descrição

## What happened

On two RTX 3090 cards, each running a **single-GPU replica with --batch 2**, v0.1.43's adaptive-overlap default reduces aggregate warm-cache throughput from **153.60 to 121.53 tok/s** on the **same immutable build** (about **20.9% slower**). Setting `STRATA_ADAPT_OVERLAP=0` restores most of the release regression: matched v0.1.42 gives155.67 tok/s.

The important additional evidence here is a **2x2 overlap ON/OFF x batch ON/OFF matrix**, two fresh starts per condition with the second pass in reverse order, fixed per-GPU HTTP endpoints, and native batch timing plus thread scheduling data. The large loss reproduces in **batch mode** on this rig; solo mode is approximately neutral across the two starts.

Related: #1975, #2010 and proposed default/logging PR #2011. This report adds an Ampere/dual-socket replica-and-batching data point and identifies a specific batch synchronization path; it is **not a layer-split report** like #2038. Please consolidate into #2010 if you prefer to track the default and batch-path issue together.

## Strata version, hardware, OS and dependencies

| Item | Measured configuration |
|---|---|
| Physical CPU | **2 x AMD EPYC 7663**,56 cores/socket,112 physical cores/224 hardware threads; AVX2/FMA, no AVX-512 |
| Virtualization | Proxmox VE 9.2.3, host kernel 7.0.6-2-pve; KVM GPU passthrough |
| VM CPU allocation | **64 vCPUs**,2 virtual sockets/NUMA nodes,32 cores/socket, no guest SMT; replica0 CPUs0-31, replica1 CPUs32-63 |
| RAM |384GiB VM allocation; guest MemTotal391,693,176KiB (about373.55GiB usable) |
| GPU | **2 x RTX 3090**,24GiB each, sm_86;350W limits; reported maximum PCIe4.0 x16 per card; native H2D probe about26.7GB/s |
| Guest OS | Ubuntu26.04 LTS, kernel 7.0.0-34-generic |
| Container userland | Ubuntu24.04.2 LTS, glibc2.39-0ubuntu8.5 |
| Driver | NVIDIA open kernel module **610.43.02**, unchanged throughout |
| CUDA toolchain | **nvcc 13.0.48** (CUDA13.0); this is the build toolkit, not the driver-supported CUDA version |
| C++ / build tools | GCC/G++ 13.3.0; CMake 4.4.3; Ninja 1.13.2 |
| Python |3.12.3; NumPy 2.5.3; Jinja2 3.1.6; official Strata HTTP server |
| Docker |29.5.3; fresh containers, network-none, read-only rootfs, CPUs0-63; only the measured container has GPU compute ownership |
| Engine/source | v0.1.42 `61b3fb5dd3f1e8ec09cf7e4e05208bc6d3c46406`; v0.1.43 `7695859aa13afb248bdb4d36f9b2e95707db079d` |
| Build | Local source build on the EPYC VM; CUDA enabled, `CMAKE_CUDA_ARCHITECTURES=86`, vision disabled; matching official engine + frontend, no engine/router patch |
| Model | **RVN-Qwen3.8-Flash-Next IQ3_S**, native expert pack; local merged GGUF `RVN-Qwen3.8-Flash-Next-IQ3_S-merged.gguf`; same GGUF/pack/tokenizer in every arm. Upstream model artifact revision was not recorded |
| Expert memory | Full53.32GiB RAM arena per replica; `cudaHostRegister PORTABLE` succeeds. No explicit resident/mmap-experts mode; no explicit hugetlb allocation, transparent huge pages requested |

The Proxmox host has Samsung 990 PRO NVMe and Crucial MX200 SSD storage; the exact backing-device mapping of the benchmark's guest volume was not separately recorded, so I do not attribute the result to a specific storage device.

### Third-party dependency provenance

The two images have **byte-identical installed third-party trees**. `setup.py` declares llama.cpp revision `3cf03257f219afbe7334045ff7c6a06ac68c627d`, but setup reuses an existing cached tree and this installation was ZIP-based, without Git metadata. I have not independently proven that the cached tree equals that declared revision; the actual tree identity is supplied instead of guessing:

- `third_party/llama.cpp`:2824 regular files, snapshotSHA256 `8e0759767192b71eec3a3b104c22df71f5a8d94d9acd20dd392d05a7c27648b0`.
- Fingerprint method: sort relative paths; append UTF-8 path plus NUL and the32-byte SHA256 of each file; SHA256 the resulting sequence; exclude symlinks, `.git` and `__pycache__`.
- v0.1.42 native binary SHA256 `8d1dfccba4b97f730eeec8eb0d4065d6f9b1226effc4df7ddbcdc2398df1c20e`.
- v0.1.43 native binary SHA256 `f4488e718ceb87806560ee05d67313d663d71c676630ef65e5eb7b2c7d9757ad`.

All overlap comparisons use the **same v0.1.43 binary and dependency snapshot**, so the switch comparison does not depend on any toolchain/dependency change between releases.

## Configuration and reproduction

Common native flags per replica (paths replaced with placeholders):

```sh
--serve --pack <IQ3_S_PACK> --native <RVN_IQ3_S_MERGED_GGUF>
--ple-gguf <RVN_IQ3_S_MERGED_GGUF> --expert-profile <Strata>/data/expert-profile.bin
--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--max-context 262144 --kv int8 --kv-resident 32768
--pool-workers 31 --batch-groups 1 --trim-stage-weights
--conversation-cache-mib 16384 --conversation-cache-slots 4
--conversation-cache-min-tokens 8192
```

`--trim-stage-weights` is inherited from the serving configuration; the native log says it is ignored without a layer split. `--batch-groups 1` stays fixed throughout and is **not** the batch ON/OFF switch.

Common environment:

```sh
STRATA_IQ_PREFETCH=4096
STRATA_ROUTE_TAIL_SKIP=0
STRATA_NAN_GUARD=1
STRATA_PIN_GUARD=1
STRATA_DECODE_TIMING=1
STRATA_ADAPT_OVERLAP=0   # or1; the only change in each overlap pair
```

No `STRATA_ADAPT_OVERLAP_RESIDENT`, `--peer-device`, layer split, pipeline windows, adapt-async, remote-expert-opt, or custom calibration. The full-arena branch is eligible for actual overlap; this is not an ineligible split/resident flag comparison. ON is explicitly confirmed by each native startup banner. OFF is asserted in the harness environment and recorded in the container launch command; v0.1.43 does not print an OFF banner.

1. Run **two unmodified official servers** inside one GPU-exclusive container. ServerA: `taskset -c0-31`, config `gpu:0`, `parallel:2`, localhost port8100. ServerB: CPUs32-63, `gpu:1`, `parallel:2`, port8101. Each config uses the common flags above. This bypasses `ReplicaRouter` dispatch variation while preserving one native engine per GPU.
2. **Batch ON:** `parallel:2`, confirmed actual `--batch 2` in both native cmdlines: four total serving slots. **Batch OFF:** `parallel:1`, actual `--batch` absent: two total serving slots, the same four requests queue.
3. Prepare24 frozen synthetic tutorial requests using Strata's exact tokenizer and chat template: five cold rounds x four distinct early-prefix prompts, plus four fixed warm prompts. Calibrate each rendered input to **8192 tokens**, output cap **256**, temperature0, seeds100/101/102/103, `enable_thinking:false`, SSE + usage. Identical request bytes are reused between every condition/version.
4. Send four requests with a barrier; requests0/1 always go to GPU0's endpoint,2/3 to GPU1. For cold rounds, verify every request has **cached_tokens=0**. Prime the four warm conversations sequentially, then perform five warm bursts; actual measured warm cache is **8185 reused +7 read =8192** for all requests in all arms.
5. Run five rounds per cold/warm phase; exclude rounds0/1, retain all raw. Order pass1: `(batch,overlap)=(1,1),(1,0),(0,0),(0,1)`; pass2: reverse. Recreate container/model state every leg. Add two identically configured v0.1.42 batch-ON controls in the same session.
6. Verify actual engine version, fixed GPU/CPU affinity, exact input/output counts, one terminal event, `finish_reason:length`, one `[DONE]`, no SSE error, loaded health, and cache shape. Record native logs, per-request wall/TTFT/cache/hashes, `/metrics`, GPU telemetry, thread `stat/schedstat`, and `numa_maps`.

**440 requests total:**400 round requests plus40 warm primes. All exact-token/SSE/cache gates passed. GPU queries and ownership/lease checks succeeded at admission; each container drained before the next. Proxy overhead and production-client traffic are excluded. Host-wide CPU load outside the VM was not recorded.

## Results

The metric below is **aggregate end-to-end throughput**,1024 generated tokens divided by wall time from first request start to last completion of a four-request burst. It includes prompt reading/cache restore and queueing; it is **not single-GPU pure decode**. Each pooled median/range contains six measured rounds (three from each fresh start).

| Version | Batch | Overlap | Cold median [range], tok/s | Warm median [range], tok/s | Warm last-request TTFT median,s |
|---|---|---|---:|---:|---:|
|0.1.42|ON|N/A|79.82 [79.29-80.45]|155.67 [150.21-158.05]|0.925|
|0.1.43|ON|1|70.75 [69.47-71.67]|121.53 [118.94-125.29]|1.174|
|0.1.43|ON|0|79.69 [79.17-80.18]|153.60 [152.42-157.92]|0.953|
|0.1.43|OFF|1|76.82 [75.00-77.31]|146.09 [141.39-152.33]|3.757|
|0.1.43|OFF|0|76.86 [76.44-77.43]|147.33 [142.93-150.96]|3.790|

With batch ON, disabling overlap gives **+12.64% cold** and **+26.39% warm**. v0.1.43 OFF is -0.17% cold/-1.33% warm relative to the matched v0.1.42 control; I do not assign a cause to that small residual with only two starts.

### All measured-round throughput, not just the winning run

- 0.1.42, batch=1, overlap=N/A:
  - Start1: cold `80.4095, 79.2907, 79.6429`; warm `155.6721, 154.8946, 150.2138` tok/s.
  - Start2: cold `79.4423, 80.0010, 80.4451`; warm `158.0451, 157.8815, 155.6731` tok/s.
- 0.1.43, batch=1, overlap=1:
  - Start1: cold `70.1768, 70.5113, 71.6731`; warm `120.8655, 125.2855, 122.1978` tok/s.
  - Start2: cold `70.9797, 71.1561, 69.4736`; warm `118.9399, 120.5190, 123.1994` tok/s.
- 0.1.43, batch=1, overlap=0:
  - Start1: cold `80.1792, 79.1716, 79.9178`; warm `153.3088, 153.1520, 157.9195` tok/s.
  - Start2: cold `79.2404, 79.4629, 79.9103`; warm `152.4179, 153.8985, 156.8489` tok/s.
- 0.1.43, batch=0, overlap=1:
  - Start1: cold `77.0298, 76.6042, 77.3094`; warm `152.3327, 151.6417, 141.3947` tok/s.
  - Start2: cold `77.0900, 74.9986, 76.4673`; warm `141.8735, 142.0446, 150.1371` tok/s.
- 0.1.43, batch=0, overlap=0:
  - Start1: cold `76.9571, 77.0904, 77.4310`; warm `149.7285, 147.1247, 142.9281` tok/s.
  - Start2: cold `76.7551, 76.6121, 76.4428`; warm `147.5323, 150.9649, 145.0707` tok/s.

## Native timing and suspected mechanism

GPU1 native batch timing, mean over six measured warm rounds:

| ms/window | v0.1.43 ON | v0.1.43 OFF | v0.1.42 |
|---|---:|---:|---:|
|Whole window|27.587|21.172|21.273|
|Adaptive wait|7.605|1.238|1.352|
|CPU experts|3.440|3.345|3.323|
|GPU-reach wait|14.543|14.652|14.635|

The whole-window difference is6.415ms and adaptive-wait difference6.367ms: almost all the measured window delta is in refill synchronization.256 windows imply about1.630s additional adaptive waiting, consistent with the native wall-time difference of about1.730s. CPU expert/GPU-reach time is close. Mean swapped-expert count is4056.8 ON versus3932.3 OFF, about3.2% more, rather than the6x change in wait.

At the **measured commit**, the source suggests why this specifically hurts batch:

- [Eligibility check](https://github.com/Niko1221/Strata/blob/7695859aa13afb248bdb4d36f9b2e95707db079d/src/program/generate.cpp#L8277-L8299) does not exclude `o.batch`; independent single-GPU replicas are eligible.
- [Adaptive copy path](https://github.com/Niko1221/Strata/blob/7695859aa13afb248bdb4d36f9b2e95707db079d/src/program/generate.cpp#L8447-L8467) inserts a per-layer `cudaLaunchHostFunc` to update the gate; the completion event follows these callbacks.
- [Batch path](https://github.com/Niko1221/Strata/blob/7695859aa13afb248bdb4d36f9b2e95707db079d/src/program/generate.cpp#L9490-L9500) still joins the adaptive helper and calls `apply_pending(true)` at the end of the window.
- [Solo path](https://github.com/Niko1221/Strata/blob/7695859aa13afb248bdb4d36f9b2e95707db079d/src/program/generate.cpp#L11860-L11875) is where pending tables are published early and copies can land while the following window proceeds.

**Inference:** the batch path can pay for the new host callback