Issues / #1413

#1413 `--batch-groups G>1` + `--batch-mtp` is silently inert: the pipelined path never calls `batch_step`, but still builds per-slot drafters

open · @shahrokhzargarpour · 0 comentarios · En GitHub

Windows

Descripción

## Summary

With `--batch-groups G>1` (the pipelined multi-slot path) and `--batch-mtp` set together:

- The pipelined branch **never calls `batch_step`**, so the window is assembled with **1 row per slot** and
  the per-slot MTP draft is **not used**.
- The per-slot drafters are **still constructed** (`verifier_max_t` is raised to 8 and
  `set_batch_graph_limit(64)` is called), so VRAM is spent on drafters that never run — **≈0.9 GB per slot**.
- **No message is printed** about either. Upstream already warns about this combination (**#1253**); this
  build (v0.1.40.1) does not.

## Environment

- Strata **v0.1.40.1** (`82f46a8c`), Windows 11 (WDDM).
- 2 GPUs, layer split, MoE model with experts in RAM, `--batch 4 --batch-mtp --spec 4`.
- Repro combination: `--batch 4 --batch-groups 2 --batch-mtp`.

## Code (references)

| What | Location |
| --- | --- |
| `piped` is true when `batch > 0 && batch_groups > 1 && n_pipe > 1` | `src/program/generate.cpp:8062` |
| The loop takes the pipelined path and **never calls `batch_step`** | `src/program/generate.cpp:8062` vs `:8186` |
| Window assembled with 1 row per slot on the pipelined path | `src/program/generate.cpp:8065-8070`, `:8143-8158` |
| `--batch-groups` gate / divisibility | `src/program/generate.cpp:3687` |
| `verifier_max_t` raised to 8, `set_batch_graph_limit(64)` | `src/program/generate.cpp:6250-6253` |
| Verify row/pos constraints | `src/core/verify.cpp:2227-2231`, `:2245-2247` |
| `batch_launch` rows | `src/core/verify.cpp:2504-2509` |

The exact line numbers above are those cited in our analysis against v0.1.40.1. **Re-confirm them against HEAD
before posting** — line references drift between revisions.

## Evidence

- Startup prints the pipelined routing line: `strata batch (pipelined…)`.
- The period summary prints the `MTP drafts accepted …` clause **only** on the non-pipelined path; under
  `--batch-groups 2` that clause is **absent**.
- Per-slot drafters are built anyway (`verifier_max_t` 8, graph limit 64) → ≈0.9 GB/slot of VRAM held for
  drafters that never draft.

## The `return 1` hole

Separately from the silent case: with `--batch N>8` + `--batch-mtp` and `N/G>8`, the window reaches

```
verify: batch rows out of range (init_slots)
```

and the engine **exits with `return 1`** — a hard stop, not a degradation. This path is **unreachable without
`--batch-mtp`**; the flag is what turns a configuration error into a fatal exit.

## Proposed fix (three parts)

1. **Warn** when `piped && batch_mtp`: print one line saying the pipelined path is 1 row/slot and the per-slot
   MTP draft will not run — matching what upstream does in **#1253**.
2. **Do not build the per-slot drafters** in that combination (skip the `verifier_max_t` / graph-limit raise),
   so no VRAM is spent on drafters that never run.
3. **Close the `return 1` hole**: replace the hard exit on `batch rows out of range` with a
   degrade-and-continue (disable `--batch-mtp` with a reason, keep serving), consistent with a
   "a flag must not end the engine" contract.

## Repro (exact)

```
strata … --batch 4 --batch-groups 2 --batch-mtp
```

Watch (a) the startup line `strata batch (pipelined…)`, and (b) the period summary: the absence of the
`MTP drafts accepted` clause.

## Not tested / open

- Diagnosed by **code reading + logs** on v0.1.40.1. No controlled A/B of the pipelined path with/without
  `--batch-mtp` was run.
- The ≈0.9 GB/slot figure comes from the drafter-construction path, not from a sampled measurement of a live run.
- Whether HEAD already prints a warning is the first thing to check (see #1253).

En el sitio

Enlaces a install, modelos, releases.