Pull requests / #2830
#2830 STRATA_MTP_ROOM_MIN — a long prompt takes the MTP drafter's 512 experts (decode-only) as ring
open · @gopinath87607 · 0 comentários · No GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAWindows
Descrição
## What this changes `STRATA_MTP_ROOM_MIN=TOKENS` (`--serve` with `--mtp`): a prompt of at least TOKENS tokens (`0`: every prompt) uses the MTP drafter's 512 routed experts as the prompt path's ring while it is read. Those are 708 MB (675 MiB) that only the **decode** reads — `record_rest` is their only reader; a prompt's passes run the draft layer's dense tensors and its K/V — so the prompt path's ring is pointed **at the block itself**: `slot i = experts_base + (i - rule) * MAXBLOB` (135 slots on a Q8_0 pack, whose largest blob is 4.98 MiB). Opening and closing a room moves the ring count alone (`Prefill::set_ring_room_active`): the block is never freed, never allocated again and never moves, so no prompt path is laid out again, the decode's captured round/step graphs stay valid (nothing is dropped), and what comes back before the first window is the decode's own bytes (`MtpDrafter::experts_reclaim`). In the measured shape's successor this is two 675 MiB DMAs around the read. Limits, refused with the reason at start: `--serve` only, `--mtp` needed, `--batch` not supported (its slot drafters share the solo drafter's weights). A layer split is fine (the room goes to the prompt path on the drafter's card), `--pipeline-windows` is fine (the room closes before the request's first window), and `--prefill-head-alias` is fine. From the Maya checkout (`project-maya`), local commits `7df369c` (`STRATA_GLM_ROOM_MIN`), `d4d0792` (the tail's slots rounded up to the largest blob) and `5ccf9a4` (the in-place shape, for the MTP config's NextN block). ## What it leaves alone Unset (the default) `mtp_room_min` is `-1` and not one line of the room runs; the prompt path is laid out exactly as before. The new code is reachable only behind that check. This is by construction, not by a measurement. ## Measurements - **The first shape was measured on our Q8_0 4-way rig (2026-10-16): it works, and it bought no time.** The room opened, the 675 MiB really went into the ring, the block came back before the first window and the decode ran unchanged — but a deeper ring was not what the prompt waited on at 3,080 or 14,859 tokens. That shape laid the prompt path out again around the block and cost **3.4–4.0 s per roomed request** on top of the read. - **The shape in this PR is the in-place one** (no rebuild, no spill allocation, block never moves). It is built and compile-clean but **its runtime has not been run yet**; its expected cost is the two 675 MiB DMAs (~0.12 s), which is an estimate, not a measurement. - `7df369c` records the room as not yet measured, with its standing baseline 89.5 s (1,123 tok/s) on its byte-identical 100,535-token read. - **No speedup is claimed here.** The knob is how you measure it: the same prompt with the variable set and unset, `PP` line against `PP` line. Expect a gain only where the ring is the limit. ## Checks - **Default path:** unset = off by construction (the parsed threshold is `-1` and every room call is behind it). - **Build:** the three host files this touches (`src/prefill/prefill.cpp`, `src/core/mtp.cpp`, `src/program/generate.cpp`) compile clean against this branch's headers with the build's own flags (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also carries the change built fully with ninja (`ninja -C build strata`, CUDA 13.2). - **HIP / SYCL:** not compiled here (no toolchain on the machine). The room uses the same CUDA API the file already used (`cudaMallocHost`/`cudaFreeHost`/`cudaMemcpy` beside its existing `cudaFree`/`cudaStreamSynchronize`). - **On a start:** the engine prints the room's threshold, MiB and slots; if the drafter's block ever moved, the first shape said so — the in-place shape cannot. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.