#5867 [Engine] Pipelined batch slots: a slot that reads alone sends its next window ahead through a group that stands idle

open · @bitaccesscomau · 1 コメント · GitHub で見る

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

本文

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

Bring the first token of a turn down for conversations that take turns: while one conversation reads its prompt and the
other stands idle, the reading slot's pipeline should carry two windows instead of one, without changing any output.

A decode window waits for its verdict before its slot's next one can start. So a slot alone in the pipelined groups has
one window in the pipeline, and each of the two stages stands idle half the time. A turn read in the slot (#5864)
ran the same way, although prompt rows are kept whatever the last stage picks: a conversation whose turn arrives while
the other one is idle - an agent waiting for its tool - read at 134 to 185 tokens/s at 177K tokens and 127 to 159 at
440K, 3.51 and 4.20 s to the first token (on `main` with the changes under this one, the read without this change).

Proposed behaviour, on by default wherever the whole-window read (`STRATA_SLOT_READ_WIDE`, #5866) is on;
`STRATA_SLOT_READ_AHEAD=0` turns it off and the read commits after its verdict again, as on #5866. It needs
`--batch` (or `"parallel"` 2 or more) on a layer split, and a slot that is alone in its group while another group
stands idle:

- A window of prompt rows alone is committed by each stage as it launches it, not after the verdict; the verdict only
  counts its rows in. A stage is free again when its stream is, so the commit has run before the slot's next window is
  staged there.
- While another group stands idle, a slot that reads alone in its group sends its next whole window through that
  group's rows behind the one in flight: two windows in the pipeline, one a stage. Nothing is allocated; the borrowed
  rows are one more layout a slot.
- A window in flight ends at the turn boundary whose checkpoint is still to take; a stop lets what is in flight land,
  and the slot keeps what it read.

Measured on 2x RTX 3090 at 250 W each (limit untouched), Swift-Qwen3.8-Flash-Next IQ3_S, split at layer 27, 1M context,
`"parallel": 2`, through `serve/server.py`, on `main` with the changes under this one: the same binary with
`STRATA_SLOT_READ_AHEAD=0` and without, 4 pairs, a fresh engine a run. Two 177K-token conversations taking turns (each
sends when the other has ended, 500-token replies) read the slot at 191 to 282 tokens/s where it read 134 to 185, the
first token came after 2.41 s instead of 3.51 (-31%) and a conversation went from 63.2 to 72.6 tokens/s from its
request to its last token (+15%). At 440K, in the three pairs where every turn was read in the slot, the first token
came after 2.90 s instead of 4.20 (-31%) and the read went from 127 to 159 tokens/s to 198 to 246; in the other pair
one run's later turns left the slots through the server's 20-second rule, so nothing of this ran there. 25 to 45 of a
turn's 55 to 94 windows went ahead. (An earlier measurement on another base gave 3.5 -> 2.2 s; one of its runs shared
an engine with the run before it, and the figures here replace it.)

Tradeoffs:
- Three files, 196 lines added and 46 removed, in the change's own commit: `src/program/generate.cpp` (the read's
  windows, the commit and the launch), a paragraph in `docs/BATCHING.md` and the new flags of `tools/slot_read_test.py`;
  two follow-up commits register the new settings and put the measured figures into that paragraph (6 lines). With
  `STRATA_SLOT_READ_AHEAD=0` the commit stays where it was.
- It needs an idle group. Two slots that read at the same moment, or a slot that reads beside one that decodes, fill the
  pipeline already (2 pairs each, the switch off and on): two 177K conversations sending their turns together got the
  first token after 4.61 s instead of 4.69, 14 of a run's 540 windows went ahead; four 32K conversations at once on
  four slots, 6.38 s instead of 6.50, all decoding 239.9 -> 244.6 tokens/s, over the round: 158.4 -> 163.5.
- Nothing runs without `--batch`: with `"parallel"` unset the replies and the engine's counts are byte-identical to
  `main` at 32K and 177K (not timed).
- Whole windows only: the last rows of a read, and the rows before a turn boundary, go through the slot's own rows.
- The read through the windows still costs about 4.1 ms a token at 177K and 4.4 at 440K. The prompt path reads a turn
  faster, but only on the main sessions and with every slot stopped: reading a turn on the slot's own sessions with the
  prompt path's kernels is the lever left for the first token (not done).
- Not run: running on AMD, Intel or Apple GPUs (their builds compile), one card, more than two stages, and more than
  one idle group to borrow from (four slots with one conversation: the first idle group is used).

Is this in scope as part of the slot read? It changes no output and adds one window behind the one in flight; the code,
tests and measurements are in a pull request opened with this issue, which links here; I will rework it or close it if
the scope should be different.

### What do you do today instead?

Nothing: on `main` a pipelined group's window carries one token of each slot and waits for its verdict before the
slot's next one starts, and a slot has no read windows of its own - a turn is admitted through the main sessions. The
changes under this one add the read in the slot (#5864) and the whole window for a slot alone in its group
(#5866); neither sends a window ahead, so each of the two stages still stands idle half the time while a slot reads
alone.

関連リンク

インストール・モデル・リリースへのサイト内リンク。