Pull requests / #2738
#2738 batch-mtp: split the solo MTP budget across batch slots — multi-GPU multi-stream tuning (agent concurrency)
open · @superlee-terry · 0 comments · View on GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Description
## What Two commits on `--batch-mtp` plus one Monitor-tab improvement, aimed at **multi-GPU (layer split) + multi-stream serving**: 1. **`batch-mtp`: proposals per slot = ceil(MTP budget / batch)** — the solo MTP budget is split across the batch slots (spec 3 + 2 slots → 2 drafts each; 3+ slots → 1). Chain-verified keep, `1 + D` rows per slot per window, and a startup note about the extra draft K/V/graph memory. 2. **`batch-mtp`: D uses the MTP budget as given** (`--mtp-max-t`, else `--spec`) — not the suffix-draft `spec+2` bump. The bumped spec is the verifier's window, not the drafter's budget; without this, `--spec 3 --batch 2` silently drafts 3 per slot. 3. **monitor: show every parallel request** — the Model state card renders one row per batch slot (reading progress / decoding tokens+tok/s+elapsed / idle-held tokens), plus solo-path, waiting-for-slot and queued counts. The single progress bar only ever showed the newest request. ## Why — the real-world scenario This is a tuning pass for **agent workloads**: connecting a local Strata endpoint to agent frameworks (Hermes, Claude Code, etc.). A large task gets split into **sub-agents that fire concurrent requests**; with the stock configuration those requests queue serially, and once `--batch` is on, `--batch-mtp` on a layer split is gated off (helper expert caches) and each slot drafts only 1 proposal per window — so MTP is effectively **disabled exactly when concurrency arrives**, and per-stream decode collapses (measured: 1 stream 74 tok/s → 2 streams ~21 tok/s each on the pipelined default). The commits make the MTP budget follow the concurrency: 1 stream keeps the full solo budget (solo path, untouched), 2 streams split it 2+2, 3 streams run 1 each inside the `kVerifyMaxT = 8` window. This is the 2×RTX 3090 / ≤3 streams corner of the design space (the window cap makes D>1 at 3 streams impossible: 3 slots × 3 rows = 9 > 8, and raising the cap touches `GMAX` in the grouped-MoE kernels). ## Measured on 2× RTX 3090, layer split auto, IQ3_S, `--batch 3 --batch-mtp --batch-groups 1 --spec 3 --vram-reserve-mib 1024` Natural-stop long-form requests (~1500-token answers, `finish=stop`, not `length`-truncated): | streams | before (pipelined default) | after (batch-mtp serial windows) | | --- | --- | --- | | 1 | 74.1 tok/s (solo) | 74.1 tok/s (solo path unchanged) | | 2 | ~42 agg | **66.4 agg** (31.8 / 36.8 per stream) | | 3 | — | **88.8 agg** (30.0 / 29.0 / 34.3) | Official `tools/strata_bench.py --clients 3` on the production instance: **82.7 tok/s aggregate, 0 errors**, TTFT p50 1.06 s / p95 1.74 s. MTP acceptance rose 59.8% → 67.2% (spec 4 → 3). Single-stream costs are within noise (agent decode 109.5 → 107.0); 4K prefill −9% is the 3 slot sessions' 1.37 GiB expert-cache tax, which is the documented `--batch` cost. D=2 at 2 streams was measured four independent times: **net zero** (acceptance decay cancels the extra draft cost) — the split formula keeps D=2 available where it helps (single-GPU builds, higher acceptance) and clamps to 1 at 3+ streams where the window demands it. ## Notes - No behaviour change without `--batch-mtp`; the solo path is untouched. - The window-cap clamp (`D ≤ 8//batch − 1`) is deliberate: 3 streams × D=2 overflows `kVerifyMaxT` and measured −19% (window rotation starves the third slot). - The Monitor rows read the existing `/metrics` `live.slots` (`slots_view()`); pure frontend, no engine ABI change.
Related on this site
Quick links to related pages.