#2007 Regression: 256K + KV streaming prefill 1046 -> 803 tok/s (-23%), decode 44.2 -> 40.5 (-8%) from 0.1.40.1 to 0.1.40.2+ on sm_75 (2x T10 layer split)

本文

**What I run**

- 2x Tesla T10 16 GB (Turing, `sm_75`), `--layer-split auto` (CUDA0 layers 0-22, CUDA1 23-47), 62 GB DDR4.
- IQ3_S pack, `--kv int8`, `--max-context 262144`, `--kv-resident 32768` (KV streaming), `--spec 4 --spec-min-p 0.5`, `--prefill auto`, `--expert-cache auto`, `--remote-expert-opt`, `--vision`.
- Expert caches hold **9682 of 24576 profiled pairs (~97.2% of the routed mass)**; `pcie_frac 0.17 / 0.30`.
- Both engines are `source: local`, built on this PC with CUDA 13.0, `-DCMAKE_CUDA_ARCHITECTURES=75`.

**What I see**

With the same frozen prompt (256,698 tokens, `prompt_token_ids` equivalent via text, `cache_n=0` cold read) and the same config:

| engine | cold prefill | warm decode (512 tokens) |
| --- | ---: | ---: |
| **v0.1.40.1** (`82f46a8`, engine 0.1.40) | **1046-1048 tok/s** | **44.16-44.18 tok/s** |
| **v0.1.43** (`7695859`, engine 0.1.43) | **801-805 tok/s** | **40.2-41.1 tok/s** |

- prefill: **-23%**, decode: **-8%**.
- Decode is measured as 512 generated tokens on a prompt whose KV is already built (`cache_n=256744`), so it is not a prompt-read artifact.
- **Alternating A/B** (new → old → new → old, rebuilding only the engine between runs, same machine, same hour): 804.9 / 1043.7 / 802.6 / 1042.5 tok/s. The two groups do not overlap; the spread inside a group is < 0.5%.
- Earlier, in a different time window, the same new engine got **1048 tok/s** once — so I do not claim it is purely a code difference in every window, but in the alternation above the version is the dominant factor.
- In the alternating run I swapped only the engine binary (`engine/strata`, 59,494,008 B for 0.1.43 vs 51,955,120 B for 0.1.40) and rebuilt it from source per step, so the JSON's `engine.build.version` field is stale (it reports the BUILD.json on disk, not the binary in use). The binary size and the rebuild timestamp are the reliable markers; I aligned each measurement with the binary swap by reconstructing the prompt-read start from `timings.prompt_ms`.

**The boundary is 0.1.40.1 → 0.1.40.2.** Everything at or after 0.1.40.2 that I tried inherits it, and 0.1.40.4's only speed change is Pascal `sm_61` decode (#1469), which does not apply to `sm_75`.

**What I ruled out on this machine** (each with one 256K run, same frozen prompt):

| candidate | test | result |
| --- | --- | --- |
| machine drift / clocks / thermals | old engine re-measured later in the same hour | 1046 tok/s again (42-50 C, no throttle) |
| `STRATA_ROUTE_TAIL_SKIP` (new default 7 on CUDA) | forced `=7`, then `=0` | `=7` is **1.4% slower** than the default; the counter shows it is active (`route tail skip: 34856 missed experts skipped`). Matches your own note that a 2-GPU split with nearly every expert cached gains nothing |
| `STRATA_PREFILL_CPU_SHARE` (new default) | `=0` | 802.8 tok/s — unchanged |
| `STRATA_MMVQ_IL` (fork F4 interleaved 2-4 column dense projections, default on) | `=0` | 804.1 tok/s — unchanged |
| prompt chunking | startup lines | both engines: `prompt chunk auto: 8192 tokens, a 384-slot ring` |
| PCIe share of missed experts | startup lines | both engines: `pcie_frac 0.17` / CUDA1 `0.30` |

So it is **not** tail-skip, not the CPU share, not F4/MMVQ-IL, not the chunk size, not the PCIe share.

**Reproduction**

1. Build `82f46a8` (v0.1.40.1) and `7695859` (v0.1.43) engines from source on an `sm_75` box (or any Turing/Ampere with a layer split).
2. Start with `--serve`, `--max-context 262144`, `--kv int8`, `--kv-resident 32768`, `--prefill auto`, `--expert-cache auto`, `--remote-expert-opt`, `--layer-split auto`, 2 GPUs.
3. Send one chat completion with a ~256K-token prompt and `max_tokens: 1`; read `timings.prompt_per_second` (must have `cache_n: 0`).
4. Then send the same prompt 5 times with `max_tokens: 512`; read `timings.predicted_per_second` (each must have `cache_n` ≈ prompt length).

My harness and raw JSON: `scripts/bench_strata_256k.py` + the four `alt2-*.json` files of the alternating run.

**What I'd like**

1. A look at what in the `0.1.40.1..0.1.40.2` range could cost ~23% of prompt speed and ~8% of decode on a **2-GPU split with ~39% of the expert pairs cached** (the cache holds 9682/24576 pairs / 97.2% of the routed mass). 242 commits there, 37 with a speed/`tok/s`/`decode`/`prefill` keyword.
2. If you can point at a switch (env var or config key) that restores the old path, that is enough for me — I do not need a fix for an older release.
3. Happy to run a `git bisect` on that range if you tell me which subsystem to watch (each step costs me ~8 minutes: engine rebuild + one 256K cold read). Tell me which metric or log line to compare and I will report per-commit numbers.

**Environment**: Linux (kernel 7.0.0), driver 580.178.04, CUDA 13.0, 2x Tesla T10 16 GB (`sm_75`), 62 GB RAM, NVMe. Engine `source: local`, built with `-DCMAKE_CUDA_ARCHITECTURES=75`.