Pull requests / #2832
#2832 serve: STRATA_WARM_PROMPT (off) reads one synthetic prompt before READY
open · @gopinath87607 · 0 コメント · GitHub で見る
Server & APIAMD / HIPNVIDIA / CUDA
本文
Title: serve: STRATA_WARM_PROMPT (off) reads one synthetic prompt before READY ## What this changes `STRATA_WARM_PROMPT=N` (`--serve`, off by default): one synthetic prompt of N tokens (32 if you ask for less) is read through the whole part chain **before the engine says READY**, so a session's first request does not pay what only a session's first prompt pays — the first chunk carries every kernel's first launch, the prompt path's own allocations and the tiers' first reads, and no later prompt pays any of it. `STRATA_GLM_WARM_PROMPT` is accepted as the same knob. The synthetic ids are spread over the vocabulary (`100 + (i * 131) % 20000`), so the routes are not one expert's. It takes the same loan a request takes (the slots the prompt path is laid out in go non-resident and are refilled from the arena afterwards), and every participant's state is zeroed before and after it, so no conversation can see it. A prompt path that cannot run prints why and the first prompt pays it; the start never fails because of it. From the Maya checkout (`project-maya`), local commit `cc58309` (2026-10-14). ## What it leaves alone Unset or `0` (the default), the whole block is skipped and the engine says READY exactly as before. Nothing outside the block changes; this is by construction. ## Measurements The Maya checkout's issue #8 reported 95–133 ms/token on a first prompt against 23–44 warm — that checkout's number on its rig. On this engine the knob is how you measure it: the first request's time to first token, or its `PP` line, with the variable set and unset is the A/B. **No number is claimed here**, because it has not been run on our hardware. ## Checks - **Default path:** unset = off by construction (`wn = 0` skips the block). - **Build:** `src/program/generate.cpp` compiles clean against this branch's headers with the build's own flags (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also carries the change built fully with ninja (CUDA 13.2). - **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so both builds compile the same change. - **On a start:** the engine prints `strata serve: warming the prompt path with a N-token prompt before READY`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
関連リンク
インストール・モデル・リリースへの站内リンク。