#1906 `--batch 2 --batch-groups auto` on a real agent client: 9 full prompt re-reads (608 s of reading) in 23 minutes, sibling group decode falls to 4-13 tok/s

说明

## Summary

Windows, 2x RTX 3080 20 GB layer split, engine **0.1.42**, `"parallel": 2` with `--batch-groups auto` (2 groups of 1 slot, i.e. the pipelined path), Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS + MTP. A real agent client keeps **two conversations** alive (48–55K tokens each) and issues two tool-call turns concurrently (`skill_manage` + `execute_code`).

In one 23-minute window the engine logged **33 prompt reads; 11 of them with `reused 0`** — the whole prompt read from token 0. Two of those 11 are negligible (one 293-token probe and one synthetic request of mine, marked below); the **9 real ones are 38K–180K tokens and cost 608.6 s of reading** (24–272 s each). Seconds earlier the *same* conversation had been re-admitted as a **copy**: `slot 0 takes 47688 tokens (copied in 456.9 ms)`. While a read runs, the other group's decode falls to **3.7–13 tok/s** (29–37 tok/s when both groups decode).

No hang and no disconnect in this window: `/metrics` showed `running: 2, queued: 0, admitting: 0` throughout, `finish: stop`. This is a *performance* report, not the #1603 hang.

## Rig, config

- Windows 11 / WDDM, **2x RTX 3080 20 GB**, layer split `auto` -> CUDA0 layers 0-24, CUDA1 25-47 (K=25, predicted 33.1 ms/window)
- Xeon E5-2696 v3 (18C/36T), **no AVX-512**, 128 GB RAM
- engine 0.1.42 from the release zip (`strata.exe` 165,004,288 B), `strata-vision.exe` 108,172,288 B
- args: `--pack <iq3_xxs> --native <shard> --ple-gguf <shard> --expert-profile <bin> --expert-cache auto --prefill auto --batch-groups auto --spec 4 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --control-vector-scaled <esp.gguf>:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --remote-expert-opt --spec-min-p 0.5 --layer-split auto --batch 2`
- `"parallel": 2`, **no `--conversation-cache-*`** configured
- start-up: `--batch-groups auto: 2 groups of 1 slot`; expert cache `8261 slots (12.04 GiB)` + CUDA1 `6566 slots (11.90 GiB)` = 14,827; slot sessions take 1.03 GiB on CUDA0; prompt path borrows 2502 CUDA0 + 2006 CUDA1 slots
- PCIe probe 12.4 GB/s -> `pcie_frac 0.34`, miss cost x1.61
- Note for #1668: `STRATA_ROUTE_TAIL_SKIP`'s default is off here because `o.batch != 0` (`src/program/generate.cpp:8076`), so this run does **not** have that 0.1.42 default either

## Evidence 1 — the re-reads (all from one session, 08:34:59–08:58)

| prompt | reused | read | read speed | generated | decode |
| --- | --- | --- | --- | --- | --- |
| 151,335 | **0** | 272.3 s | 556 tok/s | 1542 | 80.4 tok/s |
| 180,159 | **0** | 112.5 s | 1602 tok/s | 9501 | 65.3 |
| 38,350 | **0** | 23.6 s | 1622 tok/s | 1 | 38.1 |
| 42,855 | **0** | 21.8 s | 1962 tok/s | 385 | 68.6 |
| 48,273 | **0** | 24.1 s | 2000 tok/s | 499 | 63.3 |
| [40,687 | **0** | 17.3 s | 2351 tok/s | 1 | 43.2] | *synthetic, mine — excluded from the totals* |
| 51,068 | **0** | 24.4 s | 2090 tok/s | 1 | 45.1 |
| 44,528 | **0** | 22.1 s | 2010 tok/s | 1 | 21.1 |
| 54,301 | **0** | 38.8 s | 1400 tok/s | 1 | 13.5 |
| 54,569 | **0** | 68.8 s | 793 tok/s | 1 | 40.3 |

9 real re-reads, 608.6 s of reading. The other 22 admissions in that session reused: e.g. `48157 = 47697 reused + 1 read`, or `47550 = 47137 reused + 413 read`.

Give back 8x, `slot N takes` 15x in the same window. Every 1-token admission reports `drafts accepted 0 of 0` (the pipelined path, as documented in `docs/BATCHING.md`), while the longer generations had working drafts (`drafts accepted 6030 of 8284`, `suffix drafts: 343 windows, 1022 of 1029 accepted`).

## Evidence 2 — the same conversation, copy vs read, seconds apart

```
strata batch: slot 0 gave back 45097 tokens of this conversation (all it holds) in 419.4 ms
strata batch: slot 0 takes 47137 tokens (copied in 475.3 ms)
strata serve: prompt 47137 tokens = 45097 reused + 2040 read in 5695 ms (358.2 tok/s), 1 generated ...
...
strata batch: slot 0 takes 47688 tokens (copied in 456.9 ms)
strata serve: prompt 47688 tokens = 47687 reused + 1 read in 3 ms (340.8 tok/s), 1 generated ...
...
strata batch: slot 1 takes 54301 tokens (copied in 832.1 ms)
strata serve: prompt 54301 tokens = 0 reused + 54301 read in 38786 ms (1400.0 tok/s), 1 generated ...
```
The copy path is 0.46 s; the read path for a conversation of the same size is 38.8 s. Both happen with the same two conversations in the same minute.

## Evidence 3 — what the sibling group does while a read runs

Console, two requests interleaved (server's own progress lines), during a 54.5K read:

```
[strata] writing a tool call: execute_code: 152 of max 213875 tokens, 13.6 tok/s, 19 s
[strata] writing a tool call: skill_manage:  193 of max 207835 tokens,  9.1 tok/s, 61 s
[strata] reading the prompt: 16,384 of 54,569 tokens, 23 s so far
[strata] writing a tool call: execute_code: 162 of max 213875 tokens,  4.8 tok/s, 41 s
[strata] reading the prompt: 24,576 of 54,569 tokens, 32 s so far
[strata] writing a tool call: execute_code: 162 of max 213875 tokens,  3.7 tok/s, 51 s
```
`/metrics` sampled every 12 s in the same window:

| time | slot 0 | slot 1 | hw total | GPU | PCIe rx |
| --- | --- | --- | --- | --- | --- |
| 08:53:20 | decoding 28.8 tok/s | decoding 30.0 | 62.7 | 76 % | 477 MB/s |
| 08:53:45 | decoding (mean 114.7) | **reading 51,068** | 23.6 | 98 % | 35.4 GB/s |
| 08:53:57 | **gen frozen at 792 for 12 s** | still reading | 17.2 | 99 % | 3.9 GB/s |
| 08:54:09 | 31.5 | 30.5 | 61.3 | 70 % | 397 MB/s |

So two groups decoding = ~29–37 tok/s each (total ~60–75); one group reading = the other drops to 3.7–13 tok/s or pauses for 12 s and then resumes.

## Questions

1. What decides `takes ... (copied in 0.4–1.0 s)` (parked, reuse) vs `prompt N = 0 reused + N read in 24–69 s` for a conversation of the same size, in the same minute, with only two conversations and two slots? If the slot assignment moves the conversations between slots (the `pick_slot` / group choice), could the slot be picked by the prompt's prefix instead of by turn order?
2. On this engine, would `--batch-groups 1` show the same re-read rate? I can run the same 10-minute workload twice and send both tables (per-admission `reused`/`read ms` and the sibling's rate) if that helps.
3. `batch_read_piece()` returns 0 for `piped`, so the `STRATA_BATCH_DECODE_SHARE` / `STRATA_BATCH_YIELD_CHUNKS` interleave never runs on the pipelined path (this is what the second follow-up on #1603 points at). Is that intended? If it is, is there any supported way to bound how long the *other* group waits for a read — measured here at 12–39 s per read, on a workload where a turn is ~10 s long?

## Related
- #1603 — the hang (`running: 0`); not seen in this window. My two follow-ups on it are the 55.9 % hit-rate dump and the `--batch-groups auto` read-splitting analysis.
- #1604 — said "could not reproduce in v0.1.42" with 2x R9700 and three rounds of ~3.2K-token conversations. This report is the same area with 47–54K conversations and ~30 turns, and it does reproduce the re-read.
- #1413, and the new `include/strata/program/batch_groups.hpp` note that `auto` gives way to `--batch-mtp` / `--adapt-async` (not asked for here).
- #1668 — `STRATA_ROUTE_TAIL_SKIP=7` is off with `--batch` slots, so this configuration forgoes that 0.1.42 gain.
- #1740 — a real `--pipeline-windows 2` / `cudaSetDevice` stall; not this (the engine stayed alive here).