#5866 [Engine] A slot alone in its group reads its turn in whole windows (2x RTX 3090: first token -4% to -15%, round total -2.8% at 440K, same output)

closed · @bitaccesscomau · 0 comentarios · En GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

## Problem and result

With batch slots on a two-card layer split (#5864), a conversation's next turn of up to 1,024 new tokens is read in the
slot that holds the conversation, in that slot's own windows (`STRATA_SLOT_READ`), beside the other slots. That read
takes the rows a slot gets for its drafts: 5 rows a window at `--spec 3` with one slot a group, 4 with two. On upstream
`main` with the stack below this PR, two 177K-token conversations sending their turns together read each new turn (427 to
742 tokens in the Off runs below) at 116.4 tok/s a slot, 42.6 ms a window for
4.98 tokens. Prompt tokens need no draft, so nothing limits a read window to a slot's draft
rows.

While a reading slot is the only slot of its group that runs and has a window's worth of its turn left, it now takes the
group's whole window, 8 rows (`kVerifyNarrowT`). `STRATA_SLOT_READ_WIDE` is on by default: every group's window, its
hand-off rows and the rows its commit replays are then sized for 8 rows. `STRATA_SLOT_READ_WIDE=0` is the read as it
was, and the windows keep their old size. Without `--batch` (or `"parallel"` 2 or more) on a layer split none of this
runs: the default path is unchanged.

Measured on `main` (`13eedf55`) with this branch on it, on 2x RTX 3090 at 250 W, the same binary with the switch off and
on (on is the default), a fresh server and engine per run, two conversations sending their turns together (427 to 932
new tokens a turn, 900-token replies, four tasks):

- **Two 177K-token conversations, 1M context, 2 slots, 4 pairs:** the first token came
  4.98 -> 4.75 s (-4.5%; pair ratios
  1.060, 1.009, 1.068, 1.054, median 1.057, percentile bootstrap 95% interval
  1.009 to 1.068), with the turn read at
  116.4 -> 124.9 tok/s a slot (+7.3%, median pair ratio
  1.072). The total over the round is level: 116.4 -> 116.6 tok/s
  (pair ratios 0.989, 0.991, 1.015, 1.014), and 177.2 -> 177.5 while both decode.
- **Two 32K-token conversations on four slots, 256K context, 2 pairs:** first token
  4.41 -> 3.74 s (-15.1%; pair ratios
  1.217, 1.138), the read 126.8 -> 145.3 tok/s a slot
  (+14.6%), the total over the round 122.9 -> 130.7 tok/s
  (+6.4%; pair ratios 1.067, 1.060).
- **Two 440K-token conversations, 1M context, 2 slots, 2 pairs:** first token 6.20 ->
  5.96 s (-3.9%; pair ratios 1.033, 1.047), the read
  103.3 -> 109.2 tok/s a slot (+5.7%), but the total over
  the round 103.1 -> 100.2 tok/s (-2.8%; pair ratios
  1.000, 0.945) and 170.8 -> 160.9 while both decode
  (-5.8%; 0.970, 0.915). A loss: it misses the rule I set before the runs (Failed).

The gain in the first token is largest at 32K on four slots, where a reading slot's rows go from 4 to 8
(-15.1%); with two slots, where they go from 5 to 8, it is -4.5% at
177K and -3.9% at 440K (the 177K interval above covers both). The round's total follows it at
32K, is level at 177K and lower at 440K. A prompt row costs nearly the same in a wide window as in a narrow one: a read
window of 7.87 tokens took 62.6 ms against
42.6 ms for 4.98 at 177K (8.0 against 8.5 ms a token),
53.0 against 31.0 ms for 7.73 against
3.96 tokens at 32K (6.9 against 7.8 ms a token), and 72.2 against
48.2 ms for 7.88 against 4.98 at 440K (9.2
against 9.7 ms a token). So what is gained is the windows saved, not the rows. The earlier evidence of this change (on
v0.2.13.1, details below) gave +3.5% over the round at 177K in two pairs; on `main` that did not repeat
(+0.2%, four pairs). At 440K the earlier single pair was already lower over the round
(-1.8%), and the two pairs on `main` agree.

The switch is on by default because the first token comes sooner at every size measured and the round's total holds at
32K and 177K. At 440K, with the default, the round's total is -2.8% and the total while both decode
-5.8% (two pairs), and with the switch on the engine warned during the rounds that less than 512 MiB
of VRAM was free at the start of a request (Limits). Where between 177K and 440K the round's total turns lower is not measured;
`STRATA_SLOT_READ_WIDE=0` keeps the old read.

Exactness, on this branch's binary with whole windows on (the default): four slots greedy and two slots sampled against
each conversation alone, and the turn read in the slot against the main sessions' admission, greedy and sampled with a
presence penalty, all identical, with the reads in windows of up to 8 tokens. The default path (one conversation,
`"parallel"` unset) gave `main`'s replies and counts at every step of the identity lane on the tip of #5868, which
contains this change.

## Related discussion

Scope proposal: #5865.

Issues and pull requests, open and closed, were searched for duplicates on 2026-11-12, with seventeen queries built from
the change's own terms (`STRATA_SLOT_READ_WIDE`, `STRATA_SLOT_READ`, `pmtp_plan`, `pm_group_rows`, `w_rd_x`,
`kVerifyNarrowT`, `batch slots`, `pipelined groups`, `--batch-mtp`, `draft rows`, `prompt rows`, `slot read`,
`active-row windows`, `pad rows group window`, `read window rows`, `rows a slot window`, `a slot alone`). None of them
found an item that gives a lone slot of a pipelined group a window wider than the rows it needs for its own token and
drafts, and none names the switch. The items that touch the same rows, or a read beside the slots, are below.

- #4121 (open; `STRATA_PIPED_MTP`): a group's window carries each slot's token and its proposals, and its third fix
  sizes the verifiers' `max_t` to hold a group's rows. These are the rows this change widens: the drafts stay per slot,
  and a lone reading slot takes the rest of the window.
- #3794 (open) asks for those drafts in the pipelined slot groups, and #2973 (closed in a tracker reset, not rejected)
  had asked first. The rows "a slot gets for its drafts" are what this change lets prompt tokens fill.
- #4928 (open; `STRATA_SPLIT_LONG_LOOKUP`): the other change that moves a split's window rows (`kVerifyNarrowT` to
  `kVerifyLongT`, for long lookup windows), single conversation and with `--batch` refused. Both size the verifier's
  window, so the two may conflict.
- #1249 (closed) and #2527 (open, active-row windows): group windows that ran idle slots as pad rows; #1249 stops a
  window at its last active slot and #2527 keeps those slots' state. This takes an idle group's rows the other way, for
  the one slot that reads.
- #5844 (open; `STRATA_ADMIT_OVERLAP`) and #3382 (open; `STRATA_PREFILL_CPU_SHARE_BATCH`) let a read run beside
  decoding slots, on the main sessions' prompt path; this reads inside the slot, in the slot's own windows.
- #5811 (open; SYCL, `STRATA_PW_WITH_BATCH`) keeps the pipelined windows for a lone request beside batch slots on the
  SYCL backend: another "alone" path, on the solo side. The `STRATA_PW_WITH_BATCH=1` of the runs below is `main`'s, on
  CUDA.

This is the third of five pull requests. It sits on #5864's change (`9f7e09cc`), which sits on #5860 and #5862; #5868
builds on this one. Each targets `main`, so the diff here shows the commits under it until those land (12 commits on
`main`, listed under Validation). Once #5864 lands, a rebase drops the commits repeated from the pull requests under
this one.

Merge-tested against all 719 open pull requests on 2026-11-13, on this revision: 719 tested. The conflicts this branch
adds, with the commits under it included, are with #3800 (`serve/server.py`), #3957 (`src/program/generate.cpp`), #4604
(`src/program/generate.cpp`), #4928 (`src/program/generate.cpp`), #5293 (`src/program/generate.cpp`), #5364
(`serve/server.py`, `serve/test_parallel.py`), #5517 (`src/core/verify.cpp`), #5662 (`include/strata/core/mtp.hpp`,
`serve/server.py`, `serve/test_parallel.py`, `src/core/verify.cpp`, `src/program/generate.cpp`), #5726
(`include/strata/core/mtp.hpp`, `serve/server.py`, `serve/test_parallel.py`, `src/core/verify.cpp`,
`src/program/generate.cpp`), #5835 (`src/program/generate.cpp`), #5837 (`docs/MULTI_GPU.md`), #5850 (`serve/server.py`).
Among the stack's own branches, this one conflicts with #5860, #5864 and #5868 in `data/settings-register.json` and with
#5862 in `src/program/generate.cpp`, because each branch carries its own copy of the shared commits; a rebase after each
landing resolves them.

## Approach

`src/program/generate.cpp`, and the paragraph the commit adds to `docs/BATCHING.md`, which gives the figures measured
below.

- `pm_group_rows()`: while `STRATA_SLOT_READ_WIDE` is on, every pipelined group's window holds `kVerifyNarrowT` (8) rows:
  the verifiers' `max_t` (`batch_max_t`), the hand-off rows, the saved commit rows (`init_deferred_commit`),
  `hbase = 8 * group`. `pm_drafter_rows()`: a slot's drafter and its residual buffer take 8 rows, so it catches up a
  read window in one chain.
- `pmtp_plan` counts what a read window could hold beyond the slot's own rows (`w_rd_x`), and the launch gives the whole
  window to a slot that is the only one of its group that runs. Only the whole window: a layout of rows is a set of
  graphs on each stage, so a read adds one layout a slot and not one for every size in between.
- Diagnostics: the start-up line names the wide read (`strata generate: STRATA_SLOT_READ: an idle slot reads the next
  turn of the conversation it holds in its own windows, 5 tokens a window and 8 while it is the only slot of its group
  that runs (STRATA_SLOT_READ_WIDE), ...`; with the switch off, the same line without the part about 8), and each read's
  line says how many windows it took (`strata batch: slot 1 read the 485 new tokens of its conversation's turn in its own
  windows in 4058 ms (119.5 tok/s, 61 windows); it holds 176291`, from the first On run below).
- `STRATA_SLOT_READ_WIDE=0` is the read as it was, and the windows keep their old size. With `STRATA_SLOT_READ=0` the
  switch is off as well.
- The register commit adds the names of this PR and of the ones under it to `data/settings-register.json`.
  `docs/SETTINGS.md` stays as `main` has it.
- A fix to #5864's slot RAM check, `batch slots: the RAM guard counts --conversation-cache-mib, not a local variable`.
  The check counted `STRATA_PARK_PREFAULT_MIB`, a variable nothing on `main` sets; without it, at a 1M context with
  `"parallel": 4` and `--conversation-cache-mib 16384` on this machine, the check carved 4 slots of 12.38 GiB and the
  engine was killed for memory during start-up. It now counts an explicit `--conversation-cache-mib` instead, and the
  variable leaves the register.

## Validation

Revision: `dee619dc874ce140eecccacdd53de64f1c9699e2`, the tip of this branch: 12 commits on `main` `13eedf552417`.
In order: #5860's change (`1759d167`), #5862's two (`9e49bacf`, `883f362d`), #5864's (`9f7e09cc`), this change
(`f8830a92`), its register commit (`c99b672c`, `data/settings-register.json`), the RAM-check fix (`ec9f6aac`, the same
patch as #5864's `fed9794d`: `generate.cpp`, its paragraph in `docs/BATCHING.md`, the register), `e918577e`
(`docs/SETTINGS.md` back to `main`'s exact bytes), then the review fixes of #5860 (`cab319d0`: `docs/MULTI_GPU.md`,
comment wording in `generate.cpp`), #5862 (`d4fab48e`: a test case and messages in `conv_cache_test.cpp`, a row in
`docs/DETAILS.md`) and #5864 (`864081e4`: `docs/BATCHING.md`), cherry-picked with the same patches, and this branch's
own (`dee619dc`: `docs/BATCHING.md`, the table above in place of the earlier base's). The commits after `ec9f6aac`
change documentation, tests and comments only: no line of `src/` outside a `//` comment or a test. The binary of every
timing and exactness run below was built from `f8830a92`, so it predates the RAM-check fix; the CUDA base-against-head
build below is of `ec9f6aac`, the same code as this head, comments aside. The fix changes only what the slot RAM check
counts: built from `ec9f6aac` and started (no requests) without `STRATA_PARK_PREFAULT_MIB`, the measured
configurations keep their slots (1M, `"parallel": 2`: `--batch 2: the slots' K/V takes 12.38 GiB of pinned RAM each;
105.9 GiB are available, of which the engine st

En el sitio

Enlaces a install, modelos, releases.