Pull requests / #2832

#2832 serve: STRATA_WARM_PROMPT (off) reads one synthetic prompt before READY

open · @gopinath87607 · 0 comentários · No GitHub

Server & APIAMD / HIPNVIDIA / CUDA

Descrição

Title: serve: STRATA_WARM_PROMPT (off) reads one synthetic prompt before READY

## What this changes

`STRATA_WARM_PROMPT=N` (`--serve`, off by default): one synthetic prompt of N tokens (32 if you ask for less) is read
through the whole part chain **before the engine says READY**, so a session's first request does not pay what only a
session's first prompt pays — the first chunk carries every kernel's first launch, the prompt path's own allocations
and the tiers' first reads, and no later prompt pays any of it. `STRATA_GLM_WARM_PROMPT` is accepted as the same
knob.

The synthetic ids are spread over the vocabulary (`100 + (i * 131) % 20000`), so the routes are not one expert's. It
takes the same loan a request takes (the slots the prompt path is laid out in go non-resident and are refilled from
the arena afterwards), and every participant's state is zeroed before and after it, so no conversation can see it. A
prompt path that cannot run prints why and the first prompt pays it; the start never fails because of it.

From the Maya checkout (`project-maya`), local commit `cc58309` (2026-10-14).

## What it leaves alone

Unset or `0` (the default), the whole block is skipped and the engine says READY exactly as before. Nothing outside
the block changes; this is by construction.

## Measurements

The Maya checkout's issue #8 reported 95–133 ms/token on a first prompt against 23–44 warm — that checkout's number
on its rig. On this engine the knob is how you measure it: the first request's time to first token, or its `PP` line,
with the variable set and unset is the A/B. **No number is claimed here**, because it has not been run on our
hardware.

## Checks

- **Default path:** unset = off by construction (`wn = 0` skips the block).
- **Build:** `src/program/generate.cpp` compiles clean against this branch's headers with the build's own flags
  (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also
  carries the change built fully with ninja (CUDA 13.2).
- **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so both
  builds compile the same change.
- **On a start:** the engine prints `strata serve: warming the prompt path with a N-token prompt before READY`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.