#6418 [Engine] Leave STRATA_BATCH_SELF_COMMIT out of the Linux CUDA parallel defaults when STRATA_PIPED_MTP is asked (#6160)
open · @fioruccione · 0 comentários · No GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Descrição
## Problem and result On Linux CUDA with `--batch` 2 or more the engine's parallel stack sets `STRATA_BATCH_SELF_COMMIT=1`, and that turns `STRATA_PIPED_MTP` off: asking for PIPED_MTP did nothing unless you also set `STRATA_BATCH_SELF_COMMIT=0` by hand. With this change, when PIPED_MTP is asked the stack leaves that one switch out and PIPED_MTP runs. Nothing changes for a start that doesn't ask for PIPED_MTP, and a start where PIPED_MTP can't run after all gets today's stack back. On 2x RTX 4060 Ti the 20K + 3 short case goes from 50.9 to 60.4 tok/s with `"parallel": 2` and from 72.8 to 80.2 with 4, and with the cache pinned the answers one at a time are identical to today's default (details below). ## Related discussion #6160. You confirmed the cause there and asked for the 20K prompt + 3 short case on the same cards and an answers-identical test, and to open the patch as a PR. ## Approach - `parallel_default_env_cuda` / `apply_parallel_defaults_cuda` take `piped_mtp` and leave only SELF_COMMIT out of the stack. - Where PIPED_MTP is decided for good (after the fit, `generate.cpp` around 7850), if it can't run, SELF_COMMIT goes back on with `apply_self_commit_default()`. The verifiers read the switch first in `Verifier::init_slots`, well after that point. A value the user set is never touched. - One more reason PIPED_MTP stays off there: slots that draft with DFlash2 (`batch_df2_wanted`) have no MTP slot drafters. Today SELF_COMMIT=1 hides that case; without it `STRATA_BATCH_DF2=1` + PIPED_MTP on pipelined groups would reach the path that crashed in my earlier tests. - A log line either way: "STRATA_BATCH_SELF_COMMIT left off: it would turn STRATA_PIPED_MTP off" or "... back on: STRATA_PIPED_MTP does not run here". Left out on purpose, to keep this to #6160: the Windows gfx1151 table (same collision, but I can't build or run that path here), the DFlash2 slot drafts decided before the fit, the #1413 warning, the launcher profiles that set SELF_COMMIT=1 themselves, and the server's "Drafting: none in parallel slots" line. I can send those separately if you want them. ## Validation **Passed:** - `arch_defaults_test` (CPU build, this branch): OK, with new checks for the left-out switch and the put-back. Taking out either rule makes them fail (2 and 1 FAIL). - CUDA build sm_89 (CUDA 12.6) of v0.2.17 `12d491da4` with and without the change, same flags: only the usual unused-variable warnings. - Speed, Threadripper PRO 3975WX, 2x RTX 4060 Ti 16 GB, layer split auto, Qwen3.8-Flash-Next IQ3_S, `--spec 4 --batch-mtp`, int8 KV, `STRATA_PIPED_MTP=1 STRATA_PIPED_CONCURRENT=1`. Three arms, a fresh engine per start, order A B C C B A: A is v0.2.17 as is (log: "STRATA_PIPED_MTP is off: it does not combine with STRATA_BATCH_SELF_COMMIT"), B is v0.2.17 + this change (log: "left off", PIPED_MTP on), C is v0.2.17 with `STRATA_BATCH_SELF_COMMIT=0` by hand. Aggregate tok/s, median of 6 bursts (3 per start), requests in a burst start together, T=0, thinking off: | Burst | A p2 | B p2 | C p2 | A p4 | B p4 | C p4 | |---|---|---|---|---|---|---| | 2 short | 84.5 | 120.2 | 120.7 | 83.6 | 120.3 | 120.8 | | 4 short | 83.8 | 120.3 | 120.6 | 135.2 | 170.4 | 168.7 | | 20K + 1 short | 39.9 | 43.2 | 43.1 | 39.4 | 42.8 | 41.7 | | **20K + 3 short** | **50.9** | **60.4** | 65.6 | **72.8** | **80.2** | 78.5 | | slowest first token, 20K + 3 | 23.0 s | 19.9 s | 17.9 s | 12.9 s | 12.6 s | 12.9 s | The long-prompt case goes up too (+19 % on `"parallel": 2`, +10 % on 4) and the slowest first token doesn't get worse. B and C are the same within the spread. - Answers. With the box's normal settings the text isn't repeatable from one start to the next even for the same binary (A against A: 3 of 14 identical), because the expert cache comes out a few slots different each start and the GPU and CPU round the experts differently. So for this check I used the recipe from DETAILS.md (`STRATA_IQ_MT_MIN=1 --prompt-cache 0 --adapt-swaps 0 --pcie-frac 0`) and `--expert-cache 3800`, which stays under the fit and gives 5241 slots on every start. 14 texts per dump (5 short prompts and a ~20K one sent one at a time, then two bursts of 20K + 3 short), 256 tokens, two dumps per start, order A B C A B C: - one at a time: all 6 texts identical in all 12 dumps of A, B and C, the 20K one included. Today's default runs these without slot drafts and B verifies MTP drafts in the pipelined groups, and the text is the same. - bursts: the 20K requests identical in all 12. The short ones beside them have 2-3 variants, in every arm and also between two dumps of the same start (A 7 of 8 against itself, B 6 of 8), which is the concurrency effect BATCHING.md already describes. **Failed:** - none from this change. One hang on arm C (v0.2.17 with the `=0` workaround, so the same path this enables), on the first 20K read beside a decoding slot: "no progress for 60 s ... reading the prompt (batched): layer 0 of the prompt chunk from token 0", with the stall report saying "the prompt's GPU work (GPU 0) had finished, but the engine was still waiting for it". The watchdog restarted the engine. I couldn't get it again: 24 more bursts of a short decode plus a 20K read on fresh engines, B and C alternated, all fine, and nothing like it in my earlier PIPED_MTP runs on 0.2.15 and 0.2.16. I have the full log if it helps. **Not run:** - Windows / HIP / SYCL / Metal builds: not touched by the change, not built here. - Other cards, prompts longer than ~20K, `"parallel"` above 4. ## Limits One box (sm_89, two cards, IQ3_S). The Windows gfx1151 path still has the same collision.
No site
Links install, modelos, releases.