#5868 [Engine] Pipelined batch slots: a slot that reads alone sends its next window ahead (2x RTX 3090, chats taking turns: -31% first token, same output)

closed · @bitaccesscomau · 0 コメント · GitHub で見る

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

本文

## Problem and result

A decode window waits for its verdict before its slot's next one can start. So a slot that is alone in its pipelined
group has one window in the pipeline, and each of the two stages stands idle half the time. A turn read in the slot
(#5864, in whole windows since #5866) ran the same way, although prompt rows are kept whatever the last stage picks.
When two conversations take turns - an agent waits for its tool while the other one works - the slot read a turn's 427
to 742 new tokens at 134 to 185 tokens/s at 177K tokens and at 127 to 159 tokens/s at 440K, and the first token came
after 3.51 and 4.20 s (this head with the switch off, below).

With this change a read's next window starts behind the one in flight, through the rows of a group that stands idle,
and each stage commits a window of prompt rows as it launches it instead of after its verdict. Nothing else changes:
the rows are the slot's own prompt rows, and the slot keeps what it read. It is on by default wherever the whole-window
read (`STRATA_SLOT_READ_WIDE`, #5866) is on; `STRATA_SLOT_READ_AHEAD=0` turns it off, and the read then commits after
its verdict again, as on #5866. Nothing of this runs on the default path: a turn is read in a slot only with `--batch`
(`"parallel"` 2 or more) on a layer split, with the pipelined groups' drafts and their deferred commit.

Measured on `main` with this stack on 2x RTX 3090 (the binary built from this head's code), the same binary with
`STRATA_SLOT_READ_AHEAD=0` and with it unset, 4 pairs, a fresh server and engine per run, two conversations taking
turns (the second sends 11 s after the first, 13 s at 440K; 500-token replies):

- at 177K the first token came after 2.41 s instead of 3.51 (-31.3%; pair ratios 1.454, 1.464, 1.456, 1.451,
  percentile bootstrap 95% interval 1.451 to 1.464). The slot read 243.3 tokens/s instead of 161.7 (+50.4%), with 25 to
  45 of a turn's 55 to 94 windows sent ahead. A conversation's tokens/s from its request to its last token rose 14.8%
  (63.2 to 72.6). Decode, once the first token is out, moved by -1.6% (pair ratios 0.976, 1.004, 1.004, 0.953).
- at 440K, all four pairs: the first token's median pair ratio 1.444, but its interval runs from 0.905 to 1.449, so the
  rule I set before the batch did not hold at 440K (Failed). In the three pairs where every turn of both runs was read
  in the slot, the first token came after 2.90 s instead of 4.20 (-30.8%; 1.441, 1.449, 1.446) and the read went from
  147.5 to 229.0 tokens/s. In the other pair (the first) the switch-on run's second conversation came back more than
  20 s after its slot's last request, so the server (#5864's `STRATA_PARALLEL_STICKY_S`, 20 s) sent that turn and every
  later turn of the run to the solo path, where nothing of this change runs; that pair's ratio is 0.905.
- where it should change nothing - two 177K conversations sending their turns together, and four 32K conversations on
  four slots (2 pairs each) - the switch-on runs sent 14 and at most 4 windows ahead a run, and every change of the
  means stayed within 3.2% (single pair ratios 0.977 to 1.066): first token 4.69 -> 4.61 s and 6.50 -> 6.38 s, all
  conversations over the round -1.1% and +3.2%.

With `"parallel"` unset the replies and the engine's counts are byte-identical to `main` at 32K and 177K.

The first version of this pull request (and #5867) said 3.5 -> 2.2 s (-38%) at 177K. That was wrong: its second
switch-on run ran on the engine of the first (1.96 s, two turns admitted, not read in the slot). On a fresh engine the
fork build gave 3.51 and 3.53 s off and 2.41 s on, as here; the figures in this description replace those of #5867.

Against `main` (the whole stack, not this switch), one run each at 177K taking turns: first token 2.75 -> 2.15 s, a
conversation's tokens/s from request to last token 75.8 -> 73.9, decode 127.3 -> 109.0. Across sessions (not paired),
over the agent, code and copy rounds, where every run read the same 427 to 742 new tokens a turn: `main` 3.04 s (4
runs of #5864's batch), #5864 3.63 s, #5866's read (this head with the switch off) 3.52 s, this change 2.43 s. In
#5864's batch the prose round read 5 new tokens in some runs, which is why its 8-turn means there (2.60 and 3.15 s) are
lower. A lone conversation decoding in its slot stays slower than `main`'s solo path; that is #5864's limit, and this
change does not alter it.

Other machines, from the comments on #5867 (not measured by me): on 2x RTX 4060 Ti 16 GB (WSL2, engine 0.1.59,
Qwen3.8-Flash-Next Q2_0, `--batch 2`, layer split) two ~32.7K conversations taking turns got a turn's first token after
0.33 to 1.79 s. No released engine has this stack (`main` has no turn read in a slot), so that is `main`'s admission
path, not this change.

## Related discussion

Scope proposal: #5867.

Issues and pull requests, open and closed, were searched for duplicates on 2026-11-12 with queries built from the
change's own terms (`STRATA_SLOT_READ_AHEAD`, `slot read ahead`, `read ahead windows`, `windows ahead`,
`pipelined prompt read`, `window in flight`, `idle group`, `turn boundary`, `slot read wide`, `STRATA_SLOT_READ`,
`batch slots`, `prompt rows commit`, `slot alone`), and again at the end of that day for items filed since (the same
queries plus `taking turns` and `turn read slot`; the two new hits, #5881 and #5839, are a field report and an EOS
issue). No item, open or closed, names the variable, a read window sent ahead of its verdict, or borrowing an idle
group's rows for a slot's read. The nearest work is #3760 (a third decode window on three or more stages) and #3095
(pipeline windows kept for the solo path beside batch slots), both other paths.

- #3760 (open): a third decode window in flight (`--pipeline-windows 3`) on three or more stages. Another way to keep
  every stage busy, for decode windows; this keeps a read's second window and adds no stage.
- #4450 (open): two alternating ~200K agent conversations under `--batch`, where 50-100% of a turn goes on giving the
  slot's conversation back and copying it in again. The workload measured here; the loss there is the idle slot's
  release, not an idle stage, and parking is not touched here.
- #2527 (open): keeps conversations in pipelined batch slots (active-row windows, `park=`, cache-aware placement). It
  places a conversation in a group's window; this reads a slot's turn in its own windows and only adds a window when the
  slot is alone.
- #2526 (open, draft): parks an idle slot's conversation before it is reused. The same idle slot, another cause.
- #559 (closed, not merged): batch slots with a pipelined layer split - the groups and windows this fills.
- #859 (closed, not merged): pipelined verify windows for one conversation on a layer split - the two-window pipeline whose shape a
  read now uses too.
- #3095 and #5811 (open, the second its SYCL port): `STRATA_PW_WITH_BATCH=1` keeps `--pipeline-windows` for a lone
  request beside `--batch` slots. That is the solo path's overlap, which this change does not touch.
- #5147 (open) and #5149 (closed, not merged): the pipelined prompt read of the main sessions (`read_windows_pl`) and
  the drafter's placement. Another prompt-read path: the main sessions' read, not a slot's.
- #1656 (closed, not merged): `--pipeline-windows` on three or more layer-split stages. More stages; this adds a window to two.
- #3794 (open): MTP drafts in the pipelined batch-slot path - drafts inside the groups, not extra windows.

#5868 is the last of five pull requests: it sits on #5866, which sits on #5864, which sits on #5860 and #5862 (the
two rewind-keep commits here are #5862's change, the same patch). Until those land the diff shows their commits too;
the commits after #5866's are this change, its settings-register commit, the RAM-guard fix, the `docs/SETTINGS.md`
restore and the review fixes of the stack, docs and tests only (Validation).

Merge-tested against all 719 open pull requests on 2026-11-13, on this revision: 719 tested. The conflicts this branch
adds, with the commits under it included, are with #3800 (`serve/server.py`), #3957 (`src/program/generate.cpp`),
#4604 (`src/program/generate.cpp`), #4928 (`src/program/generate.cpp`), #5293 (`src/program/generate.cpp`), #5364
(`serve/server.py`, `serve/test_parallel.py`), #5517 (`src/core/verify.cpp`), #5662 (`include/strata/core/mtp.hpp`,
`serve/server.py`, `serve/test_parallel.py`, `src/core/verify.cpp`, `src/program/generate.cpp`), #5726
(`include/strata/core/mtp.hpp`, `serve/server.py`, `serve/test_parallel.py`, `src/core/verify.cpp`,
`src/program/generate.cpp`), #5835 (`src/program/generate.cpp`), #5837 (`docs/MULTI_GPU.md`), #5850
(`serve/server.py`); #5293, #5662, #5726 and #5837 come with #5860, #5835 with #5862. #3957 conflicts at the `INFO ...
batch_groups=` line, where this stack appends `slot_read=` and #3957 `park_files=`; #3800 at the admission guard and
the `BGEN` line of `serve/server.py`, both from #5864.

## Approach

The change is in `src/program/generate.cpp`, with a paragraph in `docs/BATCHING.md` and new flags in
`tools/slot_read_test.py` (3 files, 196 lines added, 46 removed).

- **The commit moves to the launch.** A window of prompt rows alone (`PGroup::forced`) is committed by each stage as it
  launches it (`batch_commit_rows` behind `batch_launch_rows` on the stage's stream), not after the verdict; the
  verdict only counts its rows in. A stage is free again when its stream is, so the commit has run before the slot's
  next window is staged there.
- **The idle group's rows are borrowed.** While another group stands idle, a slot that reads alone in its group sends
  its next whole window through that group's rows behind the one in flight (`BSlot::fly`: the rows of its read in
  flight; `pmtp_plan` and the launch start behind them). Two windows in the pipeline, one a stage. Nothing is
  allocated: the borrowed rows are one more layout a slot.
- **Where a window in flight ends.** At the turn boundary whose checkpoint is still to take; a stop lets what is in
  flight land, and the slot keeps what it read. A window only goes ahead when the slot's next one is a whole window
  (`1 + w_rd + w_rd_x == pmtp_group_cap`), and one borrowed layout is used: the first group that stands idle.
- **Scope.** `slot_read_ahead = slot_read_wide && STRATA_SLOT_READ_AHEAD != 0`, with `slot_read_wide = slot_read &&
  pm_read_wide && pmtp_group_cap > pmtp_rows_per_slot`. `slot_read` needs `--batch` with more than one pipelined group
  on a layer split, the groups' MTP drafts (`STRATA_PIPED_MTP`) with the deferred commit, and `STRATA_SLOT_READ` not `0`
  (#5864). The server asks for a turn to be read in its slot only while two or more slots had requests within
  `STRATA_PARALLEL_STICKY_S` seconds (default 20, #5864), and the engine reads it there only up to
  `STRATA_SLOT_READ_MAX` new tokens (default 1,024); a longer turn takes the prompt path. Within that, this change acts
  only for a slot that reads alone in its group while another group stands idle.
- **Diagnostics.** The start-up line `strata generate: STRATA_SLOT_READ: ...` ends in `a window ahead through a group
  that stands idle (STRATA_SLOT_READ_AHEAD)` when the switch is on, and only then. Each turn read in a slot prints
  `strata batch: slot S read the N new tokens of its conversation's turn in its own windows in T ms (R tok/s, W
  windows)`, and the `strata batch (pipelined, ...)` line after it ends with `(read in the slots: N prompt tokens, K
  turns, A windows ahead)` (new in this commit).
- **Test.** `tools/slot_read_test.py`: `--alone` (a round's turns one after the other, so each slot reads beside an
  idle group), `--later-pad`, `--expect-ahead` (the test fails when no window went ahead), and `--stop-read` (a read
  stopped on its way and its turn sent again). It compares each later turn read in its slot with the same turn admitted
  through the main sessions.

## Validation

Revision `a23b8707b248b8b06c4

関連リンク

インストール・モデル・リリースへのサイト内リンク。