改动 / #2829

#2829 prefill: STRATA_PREFILL_MAX, a per-part prompt-chunk ceiling

open · @gopinath87607 · 0 评论 · 去 GitHub 看

Server & APIAMD / HIPNVIDIA / CUDA

说明

## What this changes

`STRATA_PREFILL_MAX="8192,4096,8192,8192"` — an opt-in ceiling on the prompt chunk of each prompt-path
participant, in the split's order. Entry 0 is the first card's path (it reads the chunk's embeddings and hands
its rows on), entry *i* the *i*-th stage; `0` or an absent entry is no ceiling, and values are multiples of 256.
`STRATA_GLM_PREFILL_MAX` is accepted as the same knob.

Strata's layer split is one grid: every stage reads prompt chunks of the chain's size, so a card whose expert
cache cannot fund one pulls *every* stage down to its number (the `#448` warning). The ceiling is the other
choice: a capped stage lays out at `min(chain, its cap)` and `Prefill::run_impl`'s `c0 += m.T` loop reads each
larger hand-off in steps of its own, forwarding each sub-step, while the other stages keep the chain's chunk.
A ceiling only lowers a stage and never raises one.

From the Maya checkout (`project-maya`), local commit `ef3b6e7` (2026-10-15).

Every chunk-pricing site prices a part at its own chunk: the auto scan (`fits_one`, `ring_room`, the old
candidate list), the serve loan (`lend`), the resident-RAM lend regions, stage init (at the stage's own maximum —
`relayout` requires `chunk <= T_max`), the init retry, split-small's extra slots, the own-buffers headroom test
and the head-alias candidates. The start prints one line per capped entry:
`strata serve: prompt chunk: CUDA%d reads N tokens a step (STRATA_PREFILL_MAX N)`, with
`, inside a larger chain chunk` when the cap is below the chain's.

## What it leaves alone

Unset, not one number changes: the list is empty, `part_chunk` / `part_ceiling` return the chain chunk,
`max_head` is 0, and the new print loop runs zero times. This is by construction, not by a measurement.

## Measurements

- That local commit measured moving every part as a loss: 94,840 ms for the capped 16k/16k/8k/7k shape against
  91,718.9 ms for 13 equal chunks of 7,744, on its byte-identical 100,535-token read (an earlier nesting took
  98,568.1 ms). That is why a ceiling only lowers a stage and why this is opt-in.
- **No speedup is claimed here.** On the four-way Q8_0 rig this was written for, the start log does show the case
  the knob is for — the chain settles at 4096 because CUDA3's 690 slots cannot fund 6912
  (`WARNING: prompt chunk 4096 tokens, not 6912 ...`, `#448`) — but a binding-cap A/B
  (`STRATA_PREFILL_MAX="8192,8192,8192,2048"`, expecting the chain to rise toward CUDA0's own 6912 while CUDA3
  reads its hand-offs in 2048-token steps) has not been run yet. Numbers to follow.

## Checks

- **Default path:** unset = byte-identical by construction (empty list; every priced site returns the chain
  chunk; no arithmetic changes).
- **Build:** CUDA. `generate.cpp` is host-only code and this adds no device code; it compiles clean against this
  branch's headers with the build's own flags (`-fsyntax-only`; one pre-existing unused-variable warning in
  unrelated code). The development branch that also carries the change built fully with ninja (nvcc 13.2).
- **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so
  both builds compile the same change.
- **On a start:** the new per-participant line above; an unset-vs-set A/B of the prefill rate is the test of
  whether a cap pays on a given rig.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。