#3760 [Engine] A third decode window in flight on a layer split (`--pipeline-windows 3`, opt-in): a host-side scheduler, a slot pool, per-stage undo, an 11-output draft chain
open · @ruibeikaa · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Problem and result
On a layer split of four stages, `--pipeline-windows 2` keeps two speculation windows in flight. On four GP100 dies (two PH402 boards, compute capability 6.0) a window that starts from a verdict takes about 38 ms and a speculated window about 19 ms, and each stage spends much of a window waiting for the one ahead of it. The pipelined loop's draft chain also stops at 7 outputs, which caps what a third window could verify. Deeper speculation inside one window does not pay on this card (every extra verify row costs about 2.2-2.5 ms per stage in my measurements); a third window in flight adds no rows to any window. Today `--pipeline-windows` is clamped to 2, so `3` is accepted and ignored.
With `--pipeline-windows 3` on three or more stages, three windows are in flight: the last stage verifies the oldest window while the earlier stages run two speculated ones, and a misprediction is undone stage by stage as soon as each stage is free of its doomed window. **Measured on this branch on four GP100 dies with every expert resident in VRAM (no CPU or PCIe share)**, decode goes 57.81 -> 60.41 tok/s (+4.5%; three runs per arm; three decode prompts of 600 tokens), with every reply and every item of the session state identical. On my earlier stack, over eight decode prompts of more shapes, it was +3.5%. The gain needs a fast drafter: with the generic draft kernels it was +0.15%; with the Pascal Q8_0 draft head of #3751 (now on main) it pays (details below). Without `--pipeline-windows 3` the engine runs today's pw-2 loop with the same allocations, captures and log lines.
## Related discussion
none; the drafter it pays with is #3751, merged
## Approach
Ten commits, in order: seven for the third window, three that make the session a request leaves exact. The branch head builds.
- `mtp`: the draft chain grows to up to 11 outputs, step by step (`MtpDrafter::kChainMax`, graphs for steps 1..11), with a source-copy event the new loop waits on, and `chain_cap()` (the outputs a chain may queue). Only a caller of `set_chain_max` above 7 gets the longer chain, so the output clamp at the context end is unchanged for everyone else. A parity program (`mtp_chain_parity`, configured only when `STRATA_MTP_PARITY_BASE` names the commit before this one) checks the chain against the code it replaces.
- `pl_sched` (new `src/program/pl_sched.{hpp,cpp}`): the pipelined decode loop as a host-side state machine. It decides which window launches, waits, commits, is undone or is re-staged, and emits operations; it calls no GPU code. `pl_sched_test` (CPU only, `EXCLUDE_FROM_ALL`, built with `--target pl_sched_test`) runs it against a symbolic GPU: a recorded decode trace, random worlds, every small world with every ending, and today's pw-2 loop transcribed operation for operation.
- `generate.cpp`, the engine side: a slot pool of depth + 1 slots per stage (a verifier, a hand-off per boundary, a GDN snapshot per front stage), taken from a free list instead of `seq % 2`, so a new window never writes a hand-off that a doomed window is still reading; the extra room is kept out of the expert caches at start-up. `PlIoFns` turns each scheduler operation into the verifier / snapshot / drafter call it names. The new loop is a separate branch before today's pw-2 loop, which is not changed; it emits a verdict as the pw-2 loop does, including #879 (non-finite logits end the request), the PCIe auto step after each verdict, and the adaptive thread pinned to the aux CPUs and named.
- Modes: `--pipeline-windows 3` runs depth 3 (4 slots); `STRATA_PIPELINE_UNDO_PER_STAGE=1` runs depth 2 with the per-stage undo (3 slots); `STRATA_PIPELINE_SLOTS=k` sizes the pool. `STRATA_MTP_CHAIN_MAX` (7..11, default 11) and `STRATA_MTP_CHAIN_STOP` (default 1) tune depth 3's chain. Test hooks (the switch file's new keys, `STRATA_PIPELINE_CHECK_STATE`, forced misses) need `STRATA_PIPELINE_DEBUG=1`.
- Fallbacks to today's loop, each said once: fewer than three stages (`--pipeline-windows 3` is then 2), a pool that does not fit at start-up or whose captures fail (it shrinks back to two slots), the draft layer on the first stage, a first window merged with the prompt's tail, a scheduler configuration that fails `Sched::check`, and a chain that could not be captured at 11 outputs (it stays at 7).
- Bit-identical by construction: only the oldest window reaches the last stage and only its output is emitted; a mispredicted window is undone on every stage before its replacement reaches that stage; the per-stage undo changes when a stage acts, never the order of operations in a stage's stream.
- Default path: with `--pipeline-windows` 0..2 and none of the new switches the pool is two slots and every request runs the pw-2 loop, with the same reservations and captures. One change reaches it: the drafter's staging rows grow from 2T to max(2T, T + 11) (19 instead of 16 rows at T = 8), so later buffers in its arena move, its mapped output arrays hold 16 entries instead of T, and it keeps 12 step events instead of 8. The bytes read and written and the operations are the same (checked end to end below).
- Builds: `pl_sched.cpp` is added to the `strata` target in the main CMake (CUDA/HIP) and in `metal/CMakeLists.txt`. The scheduler is host code with no backend-specific parts; SYCL has its own `generate.cpp` and is not touched. `verify.cpp` is not touched.
- A reply that stops inside a window: a verdict committed the whole accepted window and the emission loop then cut its output at `max_new` or a stop token, so the state and `Sched::end_` ran past the reply. `PlIo::emit_count` is the emission rule; the new loop's `Sched::verdict` commits only the rows the reply keeps. `pl_sched_test` (f) is the case.
- `STRATA_STATE_CLEAN` (opt-in, default off; `4` is the one to use, 1-3 are for diagnosis): the session a request leaves does not depend on the window depth. The commit ends at the reply's last token, on the serial and pw-2 loops too; the last page's unclaimed K/V cells and the indexer's spare row are reset; the drafter cells that depend on the speculation or on an earlier session are cleaned, and a resumed turn keeps its drafter window. `STRATA_DRAFT_KV_LOG` and `STRATA_STATE_DUMP` are its diagnostics. Without the switch the serial and pw-2 loops commit exactly as main does.
#3751 is what makes depth 3 pay on GP100; it is on main behind `STRATA_MTP_Q8_SM60=1`.
## Validation
**Environment note: everything below was measured with every expert resident in VRAM on four GP100 dies** (two PH402 SKU 200 boards, 32 GB HBM2 per die, 128 GB in all; the Swift 1.5 IQ3_S pack fits whole: no CPU share, no PCIe share, no adaptive tier). A single card, three stages, or experts in RAM were not measured. HIP, SYCL and Metal were not built.
**Measured revisions.** The first group under Passed is this branch (main 870aedd9, engine 0.1.54, plus the ten commits) with only switches main has. The second group is the earlier measurement on my stack (main e88ee39d, engine 0.1.45.3, plus my opt-in GP100 changes), kept because it covers more request shapes.
Rig: four GP100 dies (application clocks 715 MHz memory / 1050 MHz graphics, 140 W cap), Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022. Build: `-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60`, ggml at the pinned 3cf03257f, unpatched. Model: Swift 1.5 IQ3_S (GGUF, dense matrices in Q8_0) with an MTP drafter. For this branch: `--serve --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.85 --mtp <drafter> --max-context 262144 --kv int8 --layer-split 12,24,36 --mtp-window 8192 --pipeline-windows 2|3 --short-read 24 --ple-io ram --conversation-cache-mib 16384 --conversation-cache-slots 4`, environment `STRATA_Q8_SM60=1 STRATA_MTP_Q8_SM60=1 STRATA_STATE_HASH=1` and nothing else. For my stack, engine arguments, the same on every arm except `--pipeline-windows`: `--serve --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.85 --mtp <drafter> --max-context 1048576 --kv int8 --layer-split 12,24,36 --mtp-window 8192 --pipeline-windows 2|3 --conversation-cache-mib 32768 --conversation-cache-slots 8 --rope-scaling yarn --rope-scale 4`. Environment, the same on every arm: `STRATA_Q8_SM60=1 STRATA_STAGE_TRIM=1 STRATA_PREFILL_PIPE=70 STRATA_MMQ_RESIDENT_SORT_NE=1 STRATA_DF_BRANCH=1 STRATA_FUSE_DECODE=1 STRATA_FUSE_WAIT=2 STRATA_SIDE_MIXER=2 STRATA_GR_UP75=1 STRATA_B6_MMVQ_ROWS=1 STRATA_ADAPT_LAG=2 STRATA_ADAPT_NOWAIT=1 STRATA_DECODE_TIMING=1`, plus the stack's `STRATA_EXPERT_NIBBLE=1` (#3455's 4-bit slots), `STRATA_SPLIT_COMMIT_ASYNC=1` (an async split commit, not posted) and `STRATA_MTP_Q8_SM60=1` (#3751).
**Passed, this branch:**
- Speed, pw 2 against pw 3, one fresh engine per run, greedy, `STRATA_STATE_CLEAN=4`: a three-turn conversation (about 3.5K tokens) and three decode prompts of 600 tokens out. Decode on those three, runs in the order 2, 3, 3, 2 and then one more of each: pw 2 57.88, 57.87, 57.67, pw 3 60.40, 60.45, 60.38 tok/s: **+4.5%**. Every reply hash and every `STRATA_STATE_HASH` item equal across the runs. (The last pair is the final commit; the commit before only moves the serial and pw-2 trim behind the switch, which is on in all of these.)
- The session a request leaves, long prompts, pw 2 against pw 3, final commit: a 30,538-token read, then two more turns, 400 tokens out each. Without `STRATA_STATE_CLEAN`: 3 of 3 replies equal, but 1 of 3 sessions (`stale` after the read; after the third turn `L` and the state that follows it, on a reply stopped by its length). With `STRATA_STATE_CLEAN=4`: 3 of 3 replies and 3 of 3 sessions equal, every item. The switch costs no decode time here: 23.30 against 23.33 s at pw 2, 22.40 against 22.33 s at pw 3, for the 1,200 tokens.
- Default path: main 870aedd9 unchanged against this branch at `--pipeline-windows 2` with none of its switches, the same six requests: every reply hash and every `STRATA_STATE_HASH` item equal.
- `pl_sched_test` (sm_60 build, Windows, CPU only, 370 s): `PASS (0 failures)`. The recorded trace, random worlds and every small world with every ending, in modes N2 (today's loop), B0 and B; today's pw-2 loop against the scheduler's depth-2 mode: 802,830 worlds, every one the same line for line. New case (f), a stop at `max_new` inside a window: 0 failures with the fix, and the pre-fix rule run on the same worlds leaves the state past the reply in 35,545 of 140,580 runs (35,291 in the other mode), so the case catches the bug.
- Build of this branch: sm_60 with `STRATA_EXPERIMENTAL_SM60=ON` (`strata pl_sched_test`), clean.
**Passed, my earlier stack (not this revision):**
- Speed, pw 2 against pw 3, one fresh engine per run, runs in the order 2, 3, 3, 2, greedy. Eight decode requests per run: five short prompts (400-512 tokens out), an 8K-token read + 300 out, a fresh 26K-token read + 400 out, an 88K-token conversation + 400 out.
- Decode, tokens out / decode time, token-weighted: pw 2 54.22 and 54.22, pw 3 56.08 and 56.15 tok/s: **+3.5%**. Per request, pw 2 -> pw 3: +3.4, +1.5, +4.4, +3.6, +4.6% (the short prompts), +4.9% (8K), +2.6% (26K), +3.1% (88K).
- Per window (from `STRATA_DECODE_TIMING`): a speculated window 19.02 -> 16.58 ms; a window that starts from a verdict 38.81 -> 37.85 ms; at pw 3, windows held behind one hit 15.19 ms, behind two 17.86 ms.
- Every reply hash the same in all four runs. Every pw 3 request logged `depth3 ... mode B slots 4`; no fallback line in any run.
- The same comparison without #3751 (generic draft kernels; one run per arm, part of a longer ladder of runs): +0.15% (51.73 -> 51.81 tok/s). There the held windows took 19.29 / 20.82 ms instead of 15.19 / 17.86: the chain runs on the last die beside the last stage's verify and reads the 276 MiB draft head every step, which eats what the third window saves. The window counts and the number of windows held behind two hits were within 1% of the runs with #3751.
- Parity, pw 2 against pw 3 (same stack, one run each): 15 requests, incSur le site
Liens install, modèles, releases.