#6160 [Engine] STRATA_PIPED_MTP is turned off on Linux CUDA by the engine's own parallel defaults (STRATA_BATCH_SELF_COMMIT=1)
open · @fioruccione · 7 comments · View on GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
**Reproduction status:** Measured locally (the code path is from reading the source, confirmed by the engine's log line) I was trying `"parallel"` on my box and `STRATA_PIPED_MTP=1` never engaged. The log said: ``` strata generate: WARNING: STRATA_PIPED_MTP is off: it does not combine with STRATA_BATCH_SELF_COMMIT (the window would keep its rows itself) ``` I never set `STRATA_BATCH_SELF_COMMIT`. On a Linux CUDA build with `--batch` 2 or more, the engine sets it itself: `generate.cpp:4200` calls `apply_parallel_defaults_cuda(true, o.batch)`, and the table in `arch_defaults.cpp:235` has `STRATA_BATCH_SELF_COMMIT=1`. Nothing changes it again before the check in `generate.cpp:7583`. So with the defaults, PIPED_MTP can't run on Linux CUDA. Setting `STRATA_BATCH_SELF_COMMIT=0` by hand fixes it, and the log then says `STRATA_PIPED_MTP: 2 pipelined groups of 1 slot draft up to 1 MTP proposal per slot each window`. If I read the history right, it's just an ordering thing. PIPED_MTP came in on 10-27 (17878ac57, #4121), and it was measured then. The parallel stack became the launcher's default on 10-30 (dd471d314), and moved into the engine on 11-07 (c9abd38a7, Linux CUDA only). Windows gfx1151 gets the same table since 8becec7d5, but I don't think PIPED_MTP matters there. **What it gives once it's on.** P620, Threadripper PRO 3975WX, 2x RTX 4060 Ti 16 GB (PCIe 4.0 x8), layer split auto 24/24, Strata 0.2.15 (engine 0.1.60), Qwen3.8-Flash-Next IQ3_S, MTP `--spec 4`, int8 KV. Parallel arms: `--batch-mtp`, `STRATA_PIPED_MTP=1`, `STRATA_PIPED_CONCURRENT=1`, `STRATA_BATCH_SELF_COMMIT=0`, no `--remote-expert-opt`. A fresh engine per arm, order serial, 2, 4, 4, 2, serial, 3 rounds each, T=0, thinking off. Requests in a burst start together; aggregate = all tokens / time until the last one ends. | Burst | serial | `"parallel": 2` | `"parallel": 4` | |---|---|---|---| | 2 short | 91 | 122 | 121 | | 3 short | 88 | 109 | 139 | | 4 short | 91 | 121 | 177 | | 20K prompt + 3 short | 56 | 59 | 81 | | ... slowest first token in that burst | 22.5 s | 20 s | 12.6 s | | one request alone | 92-105 | 90-102 | 92-99 | | expert slots (both cards) | 11,496 | 10,235 | 9,120 | | VRAM free, lowest seen | 401 MiB | 606 MiB | 442 MiB | So on this box it's +34 % with two requests and almost 2x with four, and a request on its own doesn't lose anything. Four slots take 5.5 minutes to start instead of 2-3 (all the window captures), and with `--vram-reserve-mib 800` I got one "below 512 MiB" warning per start. I'd give it a bit more reserve. One thing I did not measure is `STRATA_BATCH_SELF_COMMIT=0` on its own, without PIPED_MTP, so I can't say what it costs the plain pipeline. **Smaller things from the same runs** (`"parallel": 2`, before PIPED_MTP): - Without drafts the default pipeline lost to serial on short pairs (83-85 vs 87-99), and `--batch-groups 1` was worse (71-74). So the 0.2.7 choice of the pipeline over one group looks right on two 4060 Ti too. - `STRATA_PIPED_CONCURRENT=1` does what BATCHING.md says. With a 20K prompt read beside a short request, the short one's longest stop went from 9.7 s to 0.65 s, and the aggregate went up 3 %. `STRATA_PIPED_READ_PIECE=16384` was worse on everything (−17 %). - `--remote-expert-opt` makes no difference here (serial with and without: same numbers), which matches the code comment that it's inert on a plain layer split. **Notes that are out of date** (from reading the source, not measured): `verify.cpp:322` still calls `STRATA_BATCH_SELF_COMMIT` opt-in. `BATCHING.md:496` says PIPED_MTP is off with SELF_COMMIT on, but not that the engine turns SELF_COMMIT on for you. And `data/settings-register.json` points to `generate.cpp:7579` for the read, which is at 7583. I don't know which fix you'd prefer: leave SELF_COMMIT off when `STRATA_PIPED_MTP` is set, or just document the `=0`. I'm happy to run whatever you want on this box.
Related on this site
Quick links to related pages.