Pull requests / #2829
#2829 prefill: STRATA_PREFILL_MAX, a per-part prompt-chunk ceiling
open · @gopinath87607 · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDA
Description
## What this changes `STRATA_PREFILL_MAX="8192,4096,8192,8192"` — an opt-in ceiling on the prompt chunk of each prompt-path participant, in the split's order. Entry 0 is the first card's path (it reads the chunk's embeddings and hands its rows on), entry *i* the *i*-th stage; `0` or an absent entry is no ceiling, and values are multiples of 256. `STRATA_GLM_PREFILL_MAX` is accepted as the same knob. Strata's layer split is one grid: every stage reads prompt chunks of the chain's size, so a card whose expert cache cannot fund one pulls *every* stage down to its number (the `#448` warning). The ceiling is the other choice: a capped stage lays out at `min(chain, its cap)` and `Prefill::run_impl`'s `c0 += m.T` loop reads each larger hand-off in steps of its own, forwarding each sub-step, while the other stages keep the chain's chunk. A ceiling only lowers a stage and never raises one. From the Maya checkout (`project-maya`), local commit `ef3b6e7` (2026-10-15). Every chunk-pricing site prices a part at its own chunk: the auto scan (`fits_one`, `ring_room`, the old candidate list), the serve loan (`lend`), the resident-RAM lend regions, stage init (at the stage's own maximum — `relayout` requires `chunk <= T_max`), the init retry, split-small's extra slots, the own-buffers headroom test and the head-alias candidates. The start prints one line per capped entry: `strata serve: prompt chunk: CUDA%d reads N tokens a step (STRATA_PREFILL_MAX N)`, with `, inside a larger chain chunk` when the cap is below the chain's. ## What it leaves alone Unset, not one number changes: the list is empty, `part_chunk` / `part_ceiling` return the chain chunk, `max_head` is 0, and the new print loop runs zero times. This is by construction, not by a measurement. ## Measurements - That local commit measured moving every part as a loss: 94,840 ms for the capped 16k/16k/8k/7k shape against 91,718.9 ms for 13 equal chunks of 7,744, on its byte-identical 100,535-token read (an earlier nesting took 98,568.1 ms). That is why a ceiling only lowers a stage and why this is opt-in. - **No speedup is claimed here.** On the four-way Q8_0 rig this was written for, the start log does show the case the knob is for — the chain settles at 4096 because CUDA3's 690 slots cannot fund 6912 (`WARNING: prompt chunk 4096 tokens, not 6912 ...`, `#448`) — but a binding-cap A/B (`STRATA_PREFILL_MAX="8192,8192,8192,2048"`, expecting the chain to rise toward CUDA0's own 6912 while CUDA3 reads its hand-offs in 2048-token steps) has not been run yet. Numbers to follow. ## Checks - **Default path:** unset = byte-identical by construction (empty list; every priced site returns the chain chunk; no arithmetic changes). - **Build:** CUDA. `generate.cpp` is host-only code and this adds no device code; it compiles clean against this branch's headers with the build's own flags (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also carries the change built fully with ninja (nvcc 13.2). - **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so both builds compile the same change. - **On a start:** the new per-participant line above; an unset-vs-set A/B of the prefill rate is the test of whether a cap pays on a given rig. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.