#5867 [Engine] Pipelined batch slots: a slot that reads alone sends its next window ahead through a group that stands idle
open · @bitaccesscomau · 1 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
### Area [Engine] (speed, kernels, models, the server and API) ### What do you want to do? Bring the first token of a turn down for conversations that take turns: while one conversation reads its prompt and the other stands idle, the reading slot's pipeline should carry two windows instead of one, without changing any output. A decode window waits for its verdict before its slot's next one can start. So a slot alone in the pipelined groups has one window in the pipeline, and each of the two stages stands idle half the time. A turn read in the slot (#5864) ran the same way, although prompt rows are kept whatever the last stage picks: a conversation whose turn arrives while the other one is idle - an agent waiting for its tool - read at 134 to 185 tokens/s at 177K tokens and 127 to 159 at 440K, 3.51 and 4.20 s to the first token (on `main` with the changes under this one, the read without this change). Proposed behaviour, on by default wherever the whole-window read (`STRATA_SLOT_READ_WIDE`, #5866) is on; `STRATA_SLOT_READ_AHEAD=0` turns it off and the read commits after its verdict again, as on #5866. It needs `--batch` (or `"parallel"` 2 or more) on a layer split, and a slot that is alone in its group while another group stands idle: - A window of prompt rows alone is committed by each stage as it launches it, not after the verdict; the verdict only counts its rows in. A stage is free again when its stream is, so the commit has run before the slot's next window is staged there. - While another group stands idle, a slot that reads alone in its group sends its next whole window through that group's rows behind the one in flight: two windows in the pipeline, one a stage. Nothing is allocated; the borrowed rows are one more layout a slot. - A window in flight ends at the turn boundary whose checkpoint is still to take; a stop lets what is in flight land, and the slot keeps what it read. Measured on 2x RTX 3090 at 250 W each (limit untouched), Swift-Qwen3.8-Flash-Next IQ3_S, split at layer 27, 1M context, `"parallel": 2`, through `serve/server.py`, on `main` with the changes under this one: the same binary with `STRATA_SLOT_READ_AHEAD=0` and without, 4 pairs, a fresh engine a run. Two 177K-token conversations taking turns (each sends when the other has ended, 500-token replies) read the slot at 191 to 282 tokens/s where it read 134 to 185, the first token came after 2.41 s instead of 3.51 (-31%) and a conversation went from 63.2 to 72.6 tokens/s from its request to its last token (+15%). At 440K, in the three pairs where every turn was read in the slot, the first token came after 2.90 s instead of 4.20 (-31%) and the read went from 127 to 159 tokens/s to 198 to 246; in the other pair one run's later turns left the slots through the server's 20-second rule, so nothing of this ran there. 25 to 45 of a turn's 55 to 94 windows went ahead. (An earlier measurement on another base gave 3.5 -> 2.2 s; one of its runs shared an engine with the run before it, and the figures here replace it.) Tradeoffs: - Three files, 196 lines added and 46 removed, in the change's own commit: `src/program/generate.cpp` (the read's windows, the commit and the launch), a paragraph in `docs/BATCHING.md` and the new flags of `tools/slot_read_test.py`; two follow-up commits register the new settings and put the measured figures into that paragraph (6 lines). With `STRATA_SLOT_READ_AHEAD=0` the commit stays where it was. - It needs an idle group. Two slots that read at the same moment, or a slot that reads beside one that decodes, fill the pipeline already (2 pairs each, the switch off and on): two 177K conversations sending their turns together got the first token after 4.61 s instead of 4.69, 14 of a run's 540 windows went ahead; four 32K conversations at once on four slots, 6.38 s instead of 6.50, all decoding 239.9 -> 244.6 tokens/s, over the round: 158.4 -> 163.5. - Nothing runs without `--batch`: with `"parallel"` unset the replies and the engine's counts are byte-identical to `main` at 32K and 177K (not timed). - Whole windows only: the last rows of a read, and the rows before a turn boundary, go through the slot's own rows. - The read through the windows still costs about 4.1 ms a token at 177K and 4.4 at 440K. The prompt path reads a turn faster, but only on the main sessions and with every slot stopped: reading a turn on the slot's own sessions with the prompt path's kernels is the lever left for the first token (not done). - Not run: running on AMD, Intel or Apple GPUs (their builds compile), one card, more than two stages, and more than one idle group to borrow from (four slots with one conversation: the first idle group is used). Is this in scope as part of the slot read? It changes no output and adds one window behind the one in flight; the code, tests and measurements are in a pull request opened with this issue, which links here; I will rework it or close it if the scope should be different. ### What do you do today instead? Nothing: on `main` a pipelined group's window carries one token of each slot and waits for its verdict before the slot's next one starts, and a slot has no read windows of its own - a turn is admitted through the main sessions. The changes under this one add the read in the slot (#5864) and the whole window for a slot alone in its group (#5866); neither sends a window ahead, so each of the two stages still stands idle half the time while a slot reads alone.
Mehr auf der Site
Links zu Install, Modellen, Releases.