Issues / #1413
#1413 `--batch-groups G>1` + `--batch-mtp` is silently inert: the pipelined path never calls `batch_step`, but still builds per-slot drafters
open · @shahrokhzargarpour · 0 comentários · No GitHub
Descrição
## Summary With `--batch-groups G>1` (the pipelined multi-slot path) and `--batch-mtp` set together: - The pipelined branch **never calls `batch_step`**, so the window is assembled with **1 row per slot** and the per-slot MTP draft is **not used**. - The per-slot drafters are **still constructed** (`verifier_max_t` is raised to 8 and `set_batch_graph_limit(64)` is called), so VRAM is spent on drafters that never run — **≈0.9 GB per slot**. - **No message is printed** about either. Upstream already warns about this combination (**#1253**); this build (v0.1.40.1) does not. ## Environment - Strata **v0.1.40.1** (`82f46a8c`), Windows 11 (WDDM). - 2 GPUs, layer split, MoE model with experts in RAM, `--batch 4 --batch-mtp --spec 4`. - Repro combination: `--batch 4 --batch-groups 2 --batch-mtp`. ## Code (references) | What | Location | | --- | --- | | `piped` is true when `batch > 0 && batch_groups > 1 && n_pipe > 1` | `src/program/generate.cpp:8062` | | The loop takes the pipelined path and **never calls `batch_step`** | `src/program/generate.cpp:8062` vs `:8186` | | Window assembled with 1 row per slot on the pipelined path | `src/program/generate.cpp:8065-8070`, `:8143-8158` | | `--batch-groups` gate / divisibility | `src/program/generate.cpp:3687` | | `verifier_max_t` raised to 8, `set_batch_graph_limit(64)` | `src/program/generate.cpp:6250-6253` | | Verify row/pos constraints | `src/core/verify.cpp:2227-2231`, `:2245-2247` | | `batch_launch` rows | `src/core/verify.cpp:2504-2509` | The exact line numbers above are those cited in our analysis against v0.1.40.1. **Re-confirm them against HEAD before posting** — line references drift between revisions. ## Evidence - Startup prints the pipelined routing line: `strata batch (pipelined…)`. - The period summary prints the `MTP drafts accepted …` clause **only** on the non-pipelined path; under `--batch-groups 2` that clause is **absent**. - Per-slot drafters are built anyway (`verifier_max_t` 8, graph limit 64) → ≈0.9 GB/slot of VRAM held for drafters that never draft. ## The `return 1` hole Separately from the silent case: with `--batch N>8` + `--batch-mtp` and `N/G>8`, the window reaches ``` verify: batch rows out of range (init_slots) ``` and the engine **exits with `return 1`** — a hard stop, not a degradation. This path is **unreachable without `--batch-mtp`**; the flag is what turns a configuration error into a fatal exit. ## Proposed fix (three parts) 1. **Warn** when `piped && batch_mtp`: print one line saying the pipelined path is 1 row/slot and the per-slot MTP draft will not run — matching what upstream does in **#1253**. 2. **Do not build the per-slot drafters** in that combination (skip the `verifier_max_t` / graph-limit raise), so no VRAM is spent on drafters that never run. 3. **Close the `return 1` hole**: replace the hard exit on `batch rows out of range` with a degrade-and-continue (disable `--batch-mtp` with a reason, keep serving), consistent with a "a flag must not end the engine" contract. ## Repro (exact) ``` strata … --batch 4 --batch-groups 2 --batch-mtp ``` Watch (a) the startup line `strata batch (pipelined…)`, and (b) the period summary: the absence of the `MTP drafts accepted` clause. ## Not tested / open - Diagnosed by **code reading + logs** on v0.1.40.1. No controlled A/B of the pipelined path with/without `--batch-mtp` was run. - The ≈0.9 GB/slot figure comes from the drafter-construction path, not from a sampled measurement of a live run. - Whether HEAD already prints a warning is the first thing to check (see #1253).
No site
Links install, modelos, releases.