Pull requests / #2830

#2830 STRATA_MTP_ROOM_MIN — a long prompt takes the MTP drafter's 512 experts (decode-only) as ring

open · @gopinath87607 · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAWindows

Descripción

## What this changes

`STRATA_MTP_ROOM_MIN=TOKENS` (`--serve` with `--mtp`): a prompt of at least TOKENS tokens (`0`: every prompt) uses
the MTP drafter's 512 routed experts as the prompt path's ring while it is read. Those are 708 MB (675 MiB) that only
the **decode** reads — `record_rest` is their only reader; a prompt's passes run the draft layer's dense tensors and
its K/V — so the prompt path's ring is pointed **at the block itself**: `slot i = experts_base + (i - rule) *
MAXBLOB` (135 slots on a Q8_0 pack, whose largest blob is 4.98 MiB). Opening and closing a room moves the ring count
alone (`Prefill::set_ring_room_active`): the block is never freed, never allocated again and never moves, so no
prompt path is laid out again, the decode's captured round/step graphs stay valid (nothing is dropped), and what
comes back before the first window is the decode's own bytes (`MtpDrafter::experts_reclaim`). In the measured shape's
successor this is two 675 MiB DMAs around the read.

Limits, refused with the reason at start: `--serve` only, `--mtp` needed, `--batch` not supported (its slot drafters
share the solo drafter's weights). A layer split is fine (the room goes to the prompt path on the drafter's card),
`--pipeline-windows` is fine (the room closes before the request's first window), and `--prefill-head-alias` is fine.

From the Maya checkout (`project-maya`), local commits `7df369c` (`STRATA_GLM_ROOM_MIN`), `d4d0792` (the tail's
slots rounded up to the largest blob) and `5ccf9a4` (the in-place shape, for the MTP config's NextN block).

## What it leaves alone

Unset (the default) `mtp_room_min` is `-1` and not one line of the room runs; the prompt path is laid out exactly as
before. The new code is reachable only behind that check. This is by construction, not by a measurement.

## Measurements

- **The first shape was measured on our Q8_0 4-way rig (2026-10-16): it works, and it bought no time.** The room
  opened, the 675 MiB really went into the ring, the block came back before the first window and the decode ran
  unchanged — but a deeper ring was not what the prompt waited on at 3,080 or 14,859 tokens. That shape laid the
  prompt path out again around the block and cost **3.4–4.0 s per roomed request** on top of the read.
- **The shape in this PR is the in-place one** (no rebuild, no spill allocation, block never moves). It is built and
  compile-clean but **its runtime has not been run yet**; its expected cost is the two 675 MiB DMAs (~0.12 s), which
  is an estimate, not a measurement.
- `7df369c` records the room as not yet measured, with its standing baseline 89.5 s (1,123 tok/s) on its
  byte-identical 100,535-token read.
- **No speedup is claimed here.** The knob is how you measure it: the same prompt with the variable set and unset,
  `PP` line against `PP` line. Expect a gain only where the ring is the limit.

## Checks

- **Default path:** unset = off by construction (the parsed threshold is `-1` and every room call is behind it).
- **Build:** the three host files this touches (`src/prefill/prefill.cpp`, `src/core/mtp.cpp`,
  `src/program/generate.cpp`) compile clean against this branch's headers with the build's own flags
  (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also
  carries the change built fully with ninja (`ninja -C build strata`, CUDA 13.2).
- **HIP / SYCL:** not compiled here (no toolchain on the machine). The room uses the same CUDA API the file already
  used (`cudaMallocHost`/`cudaFreeHost`/`cudaMemcpy` beside its existing `cudaFree`/`cudaStreamSynchronize`).
- **On a start:** the engine prints the room's threshold, MiB and slots; if the drafter's block ever moved, the
  first shape said so — the in-place shape cannot.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.