#2050 batch-groups: a cold prompt read no longer pauses every stream for its whole length, longest pause 12.2 -> 4.2 s on 2x RTX 5090 (opt-in)

说明

## Title
Issue: Related: #1906 (streams stop for a whole prompt read with `--batch-groups` > 1; its question 3 asks why
`batch_read_piece` returns 0 for pipelined groups; the full re-reads it also reports are a separate matter) and #1603 (a comment there reports the same stop: a 160K read with
`--batch-groups auto` left the other slot at 0.03 tok/s; the hang in its title is a separate matter). #1504 (closed) read
in pieces on `--batch-groups 1` only and left the pipelined path out; its thread asked for the follow-up TTFT during a
45K read, measured below. Nearby: #1441 (`--prefill-pipe` chunk sizes; pieces here are whole chunks of any size).
Searched 2026-10-12 for `batch_read_piece`, `piped`, `pipelined`, `BYIELD`, `batch-groups`, `DECODE_SHARE` and `stall`.

## Motivation
I run Strata on 2x RTX 5090 (`--batch-groups 2`) as the backend for Hermes agents doing development work. Each time a new
agent joins, its first request is a long cold prompt (system prompt, tools, repository context), and every agent that is
already decoding stops until that prompt has been read. With several agents at work this happens often, so the running
streams keep stalling for seconds and their tokens per second swing between full speed and zero.

This change lets a cold prompt join smoothly: the running streams keep getting tokens while it is read, and the new
agent's first token comes a little later in exchange. Like an incremental garbage collector, it turns one long
stop-the-world pause into several short ones; the total time paused stays about the same.

## Summary
**Why it matters:** with a layer split and `--batch-groups` > 1, a prompt is read in one `sp.run`. Every decoding slot
waits for the whole prompt, `STRATA_BATCH_DECODE_SHARE` has no effect, and `BYIELD` is never taken. On one GPU the slots
already decode between a prompt's chunks, and a short request can take over a long read.

**How:** `STRATA_PIPED_READ_PIECE=N` reads the prompt in pieces of N tokens (whole chunks, 1 to 32). Each piece is one
pipelined run, so only its end drains the pipeline. At each piece's end:
- a waiting request with a prompt under half as long can take over, with the existing `BYIELD` code (the drained pipeline
  is what a give-way needs);
- the decoding slots run through the pipeline for `STRATA_BATCH_DECODE_SHARE` x the piece's time, capped and floored by
  `STRATA_PIPED_DECODE_MAX_MS` / `STRATA_PIPED_DECODE_MIN_MS`.

A read that starts right after another such read gives the slots one decode budget first. Unset: one run, as before.

**Measured** on 2x RTX 5090 (Swift 1.5 IQ3_S, split 25, `--batch 6 --batch-groups 2 --resident-experts`, k8v4, 262K,
`--kv-resident 32768`): v0.1.43 vs this change, both built here, a fresh engine per run, `STRATA_PIPED_DECODE_MAX_MS=400`.
Release and 32K: median of 5 interleaved pairs (ABBA); 16K: one run.

Four streams decoding while two cold prompts join (64K, then 32K 6 s later):

| | release | 32K pieces | 16K pieces |
|---|---:|---:|---:|
| longest pause of a stream | 12.2 s | **4.2 s** | **2.6 s** |
| total time paused per stream | 12.2 s | 13.3 s | 15.7 s |
| tokens the streams got meanwhile | 289 | **527** | **793** |
| the two prompts' TTFT | 6.9 / 6.1 s | 8.1 / 8.5 s | 10.5 / 12.0 s |
| token rate of the run | 173 tok/s | 166 tok/s | 154 tok/s |

Nothing decoding, a cold 45K prompt (#1504's test, median of 3):

| | release | 32K pieces | 16K pieces |
|---|---:|---:|---:|
| one-line question sent 1 s in: its TTFT | 5.4 s | **3.6 s** | **1.8 s** |
| the 45K prompt's TTFT | 6.0 s | 7.3 s | 8.0 s |
| the same prompt read alone | 5.7 s | 6.4 s | 7.1 s |

The 32K pause was lower in 5 of 5 pairs. Total time paused counts gaps over 0.5 s (mean of the four streams).

**Where upstream's default is better:** throughput and the long prompt's TTFT. Each piece ends with a drained pipeline:
the joining prompts' TTFT is 27% longer with 32K pieces (73% with 16K), the token rate is 4% lower (within run-to-run
spread), and an idle 45K read takes 12% longer (25% with 16K), since pieces are also taken when nothing decodes. Leave it
off when aggregate tokens or one client's long prompts matter most; turn it on when a multi-second stall is worse than a
later first token. Single-chunk pieces would cost the read the most (#1504's "65% longer read").

## What changed
- `include/strata/program/batch_read.hpp` (header-only): `piped_read_piece()`, `piped_decode_budget_ms()`,
  `piped_read_lead_in()`.
- `src/program/generate.cpp`, `read_part`: the piecewise read on the pipelined path; at each piece's end the existing
  `BSTOP` / `BYIELD` handling, then `pump(true)` for the budget and `pipe_drain()`; the lead-in; one stderr line per piece
  and lead-in. Unset: `batch_read_piece()` as before.
- `tests/program/batch_read_test.cpp`: each new function (rounding, the 32-chunk cap, chunk 0, cap and floor, lead-in).
- `docs/BATCHING.md`: the opt-in's section, and how to run `batch_interleave_test.py` on pipelined groups.
- `tools/piped_read_join_bench.py` (new, stdlib only): the scenarios above (`--scenarios two_joiners,followup,read_alone
  --repeat N`) and `--determinism`.

## Extra Notes
**Text equality:** upstream's `tools/batch_interleave_test.py` passes every step with 16K pieces: C's 40K read goes in
three pieces while A and B decode, D gives way at 16,384 tokens (`YIELDED 2 16384`) while E is admitted, and every request
equals its solo tokens. The release fails step 4 (D never gives way). Settings: `--long 40000 --pcie-frac 0
--adapt-every 1000000 --no-prefill-borrow`, plus `--batch 4 --batch-groups 2` (3 slots make one group) and no
conversation cache (it would restore D in step 4). With the variables unset, both builds gave the same SHA-256 for six
greedy answers, twice each, under `--prompt-cache 0 --adapt-swaps 0 --pcie-frac 0` (DETAILS.md), and ran at the same
speed (45K read alone 5.66 vs 5.69 s, median of 3; two-joiner pause 12.2 s on both). Text equality, not bit-exactness, and
greedy only.

**Run-to-run spread:** the longest pause was 12.16-12.23 s for the release in all five runs and 4.19-4.22 s for 32K in
four. In the fifth, the 64K prompt's first piece took 9.3 s instead of 3.95 s before any slot decoded (so not this
change's code), giving a 9.5 s pause and 13.5 / 13.9 s TTFTs; it stays in the medians. Token rate ranged 169-180 tok/s
(release) and 136-174 tok/s (32K).

**Built** on one Windows 11 PC: CUDA 13.0 (VS 2022, `-DCMAKE_CUDA_ARCHITECTURES=120 -DSTRATA_PORTABLE=ON`) and HIP via
`tools\hip\build_windows.bat tests` (ROCm 10.2.0a20260930, gfx1100). `batch_read_test` passes on both and
`batch_groups_test` on HIP; the only new warnings are the MSVC `getenv` notes the file's other `std::getenv` calls also
print. SYCL is untouched: it builds its own `sycl/src/program/generate.cpp` and never includes `batch_read.hpp`. One
commit on v0.1.43 (`7695859`).