#2936 [sm_86 / Xeon E5-2680 v4] 0.1.45: decode -20-27% vs 0.1.43 on a single RTX 3090; build pins and CPU expert pool both ruled out — points at 0.1.45 source changes

Description

# [sm_86 / Xeon E5-2680 v4] 0.1.45: decode -20-27% vs 0.1.43 on a single RTX 3090; build pins and CPU expert pool both ruled out — points at 0.1.45 source changes

## Setup

- Single RTX 3090 24 GB, 88 GB RAM, expert arena fully pinned (zero disk reads)
- CPU: **Intel Xeon E5-2680 v4 @ 2.40 GHz** (12C/24T, 2016; AVX2 + FMA + F16C, **no AVX-512 / no AMX**)
- Driver 580.178.04; 350 W power limit
- Toolchain, identical for all four arms: CUDA 13.0 (`/usr/local/cuda-13.0`), gcc 13.3, `-DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=86`, ninja
- **llama.cpp submodule identical for all four arms** (one checkout, commit `296b59d`)
- Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S, 262K context, KV int8, `--spec 4 --spec-min-p 0.5 --mtp rt --ple-io ram --kv-resident 32768`
- Strata source: 0.1.43 at `e568e70371a3d2e0` (built by the repo's own `setup.py` flow — the production binary), 0.1.45 at `477ac13` (tag `v0.1.45`)
- Bench: same prompts, 5 runs x 3 warmups, narrative ~600-1200 tok and code ~450-800 tok; engine swapped between arms, service restarted, ready-poll + smoke test each time

## Observation: four arms, A/B/A'/D

| arm | Strata | Q6K_EXPERTS / MMQ_KQUANTS | narrative decode | code decode | CPU pool / expert | GPU W |
|---|---|---|---|---|---|---|
| A  | 0.1.43 | OFF / OFF (production build) | 102.6 | 118.4 | 82 us | 182 |
| A' | 0.1.43 (retest) | OFF / OFF | 103.2 | 115.9 | 82 us | - |
| B  | 0.1.45 | ON / ON (release pins) | **78.6 (-23%)** | **96.6 (-18%)** | 96-102 us | 166 |
| D  | 0.1.45 | OFF / OFF (rebuilt to match A exactly) | **79.4 (-23%)** | **86.3 (-27%)** | 82-85 us | - |

A vs A' agree within 1%, so positional/thermal bias is not the cause.

## What is ruled out

1. **Build pins.** Arm D is 0.1.45 compiled with `Q6K_EXPERTS=OFF` and `MMQ_KQUANTS=OFF` — the same cmake configuration as the 0.1.43 production binary (verified against `setup.py`: `host_compiler_defs` / `toolkit_root_defs` / `engine_defs` are all identical for sm_86 + CUDA 13; no `STRATA_EXPERIMENTAL_SM60`, no `STRATA_ISA_FLOOR` on an AVX2 CPU). Arm D still regresses, and its CPU pool reads 82-85 us/expert — the same as 0.1.43. **The pins are not the cause.**
2. **CPU expert pool throughput / PCIe.** Arm D's "PCIe share" probe reports 82-85 us per missed expert (24.0-24.8 GB/s of expert bytes), identical to 0.1.43, and the PCIe probe reads 11.3 GB/s host->device on every arm. Yet decode is still 20-27% slower. **The per-expert CPU cost is not the cause either.**

## What that leaves

With toolchain, submodule, config, argv and PCIe all matched, the delta is **Strata source between `e568e70371a3d2e0` (0.1.43) and `477ac13` (0.1.45)**. Prime suspects from the 0.1.44/0.1.45 release notes, all CPU/decode-scheduling side:

- "the pipelined-decode and latency work" (0.1.44)
- `STRATA_ROUTE_TAIL_SKIP=7` default on CUDA (0.1.42) interacting with the newer scheduler
- the MTP router guard (0.1.40.3) under `--mtp rt`
- `pcie_frac` self-adaptation converging to a different point (0.20 on 0.1.43 vs 0.20-0.25 on 0.1.45) even though the measured pool speed is the same

Corroborating signals on the regressed arms: GPU power drops 182 W -> 166 W (the GPU waits, it does not compute slower) while host CPU busy rises 50.2% -> 56.9% for the same token count.

## Related reports

- #2932 (gfx906, 2x Radeon Pro VII): 0.1.45 vs 0.1.43.1 — 32k decode ~13% slower **and 12/12 greedy texts differ**; `STRATA_TOPK_OLD=1` gives a third set of texts.
- #2711 (Strix Halo, 0.1.41): default CPU pool workers steal package power from the GPU; decode +14-30% with fewer workers. Same "CPU busier, GPU starved" shape.
- 0.1.45 release notes: "Moved to 0.1.46: the Volta/Pascal series. Together it made default decode on an RTX 3060 about 4% slower after a long prompt; we are finding the cause before it ships."

Happy to bisect `e568e70371a3d2e0..477ac13`, or re-run with `STRATA_ROUTE_TAIL_SKIP=0`, `--mtp` off, or any specific switch on this box.