#3819 [Engine] Layer split: a stage's commit without the host wait, as on one GPU (`STRATA_SPLIT_COMMIT_ASYNC=1`, opt-in)
open · @ruibeikaa · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Problem and result **Everything below was measured on four GP100 dies (two PH402 SKU 200 boards, 32 GB HBM2 per die, 128 GB in all) with every expert resident in VRAM: no CPU share, no PCIe share, no adaptive tier.** A rig that keeps experts in RAM or streams them over PCIe was not measured. HIP, SYCL and Metal were not built. On one GPU, `Verifier::commit` launches the window's commit graph and returns: the next window runs behind it on the same stream, and `wait_commit` covers the readers that need the state (`set_commit_async(true)`). On a layer split, `set_commit_async(!multi_gpu)` and the `|| next_ != nullptr` in `commit()` keep a `cudaStreamSynchronize` after every stage's commit graph, so the host waits for each stage's commit in turn before the draft. The split was left synchronous because it was untested there (1330ed56), not for a known hazard. With `STRATA_SPLIT_COMMIT_ASYNC=1`, a split's stages commit without the host wait, as one GPU already does: the same graphs on the same streams, only the host waits change, so the replies and the session state are the same bits. Unset (the default), the two calls get the values they get today. Where it applies: windows that go through `Verifier::run` and `commit` on a layer split, i.e. serial decode (no `--pipeline-windows`, or a request that `--pipeline-windows` decodes serially) and the window reads at the end of a prompt. Pipelined windows commit through `pl_commit_async` and are not affected. - Serial decode, four GP100 dies, `--layer-split 12,24,36`, measured on an older 0.1.41-based tree: the commit at the end of a window took 0.34-0.71 ms with the host wait and 0.03-0.06 ms without; every reply the same. - On main e88ee39d with `--pipeline-windows 3` on (so only the prompt-end windows take this path): every reply and every engine state hash the same on 15 requests; decode moved within the run-to-run spread, so I do not claim a speedup from that run. ## Related discussion None (an opt-in scheduling switch, no default change). This branch is main plus one commit and needs none of my other PRs. The main e88ee39d run below was on a branch that also carried #3455 (`STRATA_EXPERT_NIBBLE=1`) and #3760 (`--pipeline-windows 3`), both on in the arm before this one. ## Approach - `Verifier::set_split_commit_async(bool)` (new; `set_commit_async` is unchanged). `commit()` skips the host wait on a split stage only when both are on. - `generate.cpp`: `split_async = STRATA_SPLIT_COMMIT_ASYNC=1 && o.batch <= 0`; then `set_commit_async(!multi_gpu || split_async)` and `set_split_commit_async(split_async)`. `STRATA_COMMIT_SYNC=1` still makes every commit wait. - Why it holds on a split: - a stage's commit graph writes only its own layers' GDN state and conv tail and the indexer's tail / dead / pooled / block positions (plus the PLE history on the stage with layer 1), from buffers its own window wrote. It reads no hand-off and writes neither the hand-offs nor the final rows, so neither another stage nor the drafter touches what it writes; - its next window runs behind it on the same stream, and `run()` synchronizes that stream before the next stage starts, so `h_commit_` is rewritten only after the graph has read it (the one-GPU argument, per stage); - each stage records its own `commit_done_` and `wait_commit` already walks `next_`, so the existing waits (end of the serve decode loop, end of a window prompt read, start of each request, end of a generate run) cover every stage. Session files refuse a layer split; checkpoints, parking and slot copies synchronize each stage's device first. - Not with `--batch`: its admissions and `STRATA_PIPED_CONCURRENT` read the session from other threads and streams on paths this argument does not cover, so the switch is ignored there. - No new allocation, graph, kernel or log line. HIP builds the same host code; SYCL has its own `generate.cpp` and verifier (untouched); Metal compiles the shared verifier, where only the setter is new (its `commit()` returns through the Metal engine first). ## Validation Rig: four GP100 dies (compute capability 6.0, application clocks 1050 MHz graphics / 715 MHz memory, 140 W cap), Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022. Build `-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60`, ggml at the pinned 3cf03257f, unpatched. Model: Swift 1.5 IQ3_S (abliterated GSQ-RCO GGUF, dense matrices in Q8_0). **Passed:** - Older tree (0.1.41-based, my GP100 branch of that time), serial decode, `--layer-split 12,24,36`, one engine start per arm, without and with this change (that pair also carried the MTP drafter change of #3751 and a Volta-only upstream kernel change, neither of which touches the commit): the `commit/emit` part of the `STRATA_DECODE_TIMING` line went from 0.34-0.71 ms a window to 0.03-0.06 ms; all 10 replies the same, including a 300K-token conversation parked and restored. - main e88ee39d (engine 0.1.45.3) plus my GP100 stack, one exe for every arm, one engine start per arm, greedy; a ladder in which each arm adds one switch to the arm before. Arm S2 = #3455 + #3760 on; arm S3 = S2 + `STRATA_SPLIT_COMMIT_ASYNC=1`. - Engine arguments, the same in every arm: `--serve --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.85 --mtp <drafter> --max-context 1048576 --kv int8 --layer-split 12,24,36 --mtp-window 8192 --pipeline-windows 3 --conversation-cache-mib 32768 --conversation-cache-slots 8 --rope-scaling yarn --rope-scale 4`. Environment, the same in every arm: `STRATA_Q8_SM60=1 STRATA_STAGE_TRIM=1 STRATA_PREFILL_PIPE=70 STRATA_MMQ_RESIDENT_SORT_NE=1 STRATA_DF_BRANCH=1 STRATA_FUSE_DECODE=1 STRATA_FUSE_WAIT=2 STRATA_SIDE_MIXER=2 STRATA_GR_UP75=1 STRATA_B6_MMVQ_ROWS=1 STRATA_ADAPT_LAG=2 STRATA_ADAPT_NOWAIT=1 STRATA_DECODE_TIMING=1`. Each start's switches were checked in the engine process's environment. - Requests: five short prompts (77-82 tokens in, 400-512 out), a 9.8K-token document + 300 out, a fresh 26K-token read + 400 out and an 88K-token append to it (115K context) + 400 out - the eight benchmark prompts, decode speed = their tokens out / decode time - then prompt reads of 240 / 1.15K / 7.7K / 33K tokens and a 26K-token conversation with a 1.1K-token append (16 out each). A parity pass with `STRATA_STATE_HASH=1` and a 250K-token read added (15 requests), and a timing pass without debug switches (14 requests). - Parity, S3 against S2: every reply hash and every `STRATA_STATE_HASH` item (GDN, PLE, tail, pooled, K/V, the drafter's K/V, stale cells) equal on all 15 requests; S3's replies equal the unchanged engine's. - Decode, eight prompts token-weighted: 51.81 -> 52.14 tok/s in the timing pass, 52.01 -> 52.12 in the parity pass. The unchanged engine ran twice in the same timing pass, 47.21 and 47.33 tok/s; this difference is of that size and is not claimed. Prompt reads within -1.5..+0.9%, peak VRAM per die unchanged (25,427 / 25,113 / 25,789 / 29,227 MiB), start-up 280 -> 283 s. - This branch, rebased onto main 5c8dd322: sm_60 with `STRATA_EXPERIMENTAL_SM60=ON` (`strata`) and sm_86 without it (`strata`), both built without errors. **Failed:** none observed. **Not run:** - HIP, SYCL and Metal builds (no such machine here). - A GPU run of this branch on main 5c8dd322: the numbers above are from the revisions named there; the rebased branch was only built. - Serial decode on main: on e88ee39d every arm had `--pipeline-windows 3`, so the async split commit was reached only by the windows that read a prompt's last tokens. Serial decode with it was run on the older tree only. - A split of two or three stages; `--batch` (the switch is ignored there by design). - A data race of this kind shows up as different bits, not as an error. The evidence is the identical replies and state hashes above, not a proof over every path. ## Limits - Opt-in. `--batch` keeps the synchronous split commit. - It saves a host round trip per stage and window in serial decode on a layer split; it does nothing for pipelined windows, which already commit through `pl_commit_async`. **CI note:** the red `metal-link` check is not caused by this PR. The job fails while compiling upstream's `metal/src/mtl_rt.mm` (lines 69 and 92: `MTLGPUFamilyMetal4`, `MTLCompileOptions.mathMode`, `MTLMathModeFast` / `MTLMathModeSafe`), which the `macos-14` runner's SDK does not provide. The workflow runs on any pull request that touches `src/` or `include/`; this PR changes no file under `metal/`.
Sur le site
Liens install, modèles, releases.