Pull requests / #3309

#3309 [Engine] warm up the prompt path once before READY

open · @gopinath87607 · 0 Kommentare · Auf GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux

Beschreibung

> **Why this PR was recreated.** The original (#2832) was opened from the `gopinath87607/Strata` fork. Deleting that fork deleted its branches, and GitHub closes every pull request whose branch lived in a deleted fork - a closed PR whose source branch is gone cannot be reopened. This is the same work from a new fork, rebased onto current `main` (`e88ee39d`); the conflict with upstream's `chunk_vram` prompt planner in `generate.cpp` was resolved. It replaces the closed #2832, which was closed by the fork's deletion, not by a maintainer.

## Problem and result

A session's first prompt pays what no later prompt pays: the first chunk carries every kernel's first launch, the prompt path's own allocations and the tiers' first reads. After this change, `STRATA_WARM_PROMPT=N` (off by default) reads one synthetic N-token prompt through the whole part chain **before the engine says READY**, so the first real request does not pay it. Unset or `0`, the block is skipped and the engine says READY exactly as before.

## Related discussion

The original pull request and its discussion: #2832 (closed when its fork was deleted; this PR replaces it). The work comes from the Maya checkout (`project-maya`), local commit `cc58309` (2026-10-14).

## Approach

`STRATA_WARM_PROMPT=N` (`--serve`): one synthetic prompt of N tokens (32 if you ask for less) is read through the whole part chain before READY. `STRATA_GLM_WARM_PROMPT` is accepted as the same knob.

The synthetic ids are spread over the vocabulary (`100 + (i * 131) % 20000`), so the routes are not one expert's. It takes the same loan a request takes (the slots the prompt path is laid out in go non-resident and are refilled from the arena afterwards), and every participant's state is zeroed before and after it, so no conversation can see it. A prompt path that cannot run prints why and the first prompt pays it; the start never fails because of it.

Unset or `0` (the default), the whole block is skipped and the engine says READY exactly as before. Nothing outside the block changes; this is by construction.

## Validation

Tested revision: `af867e73`, one commit on current `main` (`e88ee39d`). Environment: Linux, CUDA 13.2 (V13.2.86), `sm_86` + `sm_120`; the build machine is the Q8_0 4-way rig (2x RTX 3060 12 GB, 2x RTX 5060 8 GB).

**Passed:**
- `git rebase e88ee39d` - the conflict in `generate.cpp` with upstream's `chunk_vram` prompt planner was resolved; the branch's diff is this feature only.
- `ninja -C build strata` at `af867e73` - exit 0 (CUDA 13.2, `sm_86` + `sm_120`).
- `-fsyntax-only` on `src/program/generate.cpp` with the build's own flags - clean; one pre-existing unused-variable warning in unrelated code. This was run before the rebase; the full build above covers the rebased head.
- Reported by blange48 on 4x RTX 5080 16 GB (2026-10-17; not run by us; Linux, layer split 12,24,36, Qwen3.8-Flash-Next IQ3_S, `--batch 8 --batch-groups 4`, v0.1.45 + this PR + #2673, one run per case, warm page cache): time to READY 102 s unset vs 101 s at `STRATA_WARM_PROMPT=512`; first request (a ~325-token prompt) time to first token 1.00 s unset vs 0.72 s at 512; later requests were unaffected.
- Maya checkout's issue #8 reported 95-133 ms/token on a first prompt against 23-44 warm - that checkout's number on its rig.

**Failed:**
- none observed.

**Not run:**
- Our own hardware: the reported run above is not ours.
- More than one run per case, and a start with a cold page cache.
- Serving the rebased head: only the build was re-run after the rebase; no GPU run was made at `af867e73`.
- HIP and SYCL builds: no toolchain on this machine. `generate.cpp` carries no per-backend code, so both builds compile the same change.

## Limits

- The report is one run per case, so its 0.28 s first-token gain is not a distribution.
- The reported build also carried #2673, whose different-greedy-answers issue is being looked at there; this PR was off in those comparisons, so it does not look involved.
- On this engine the knob is how you measure it: the first request's time to first token, or its `PP` line, with the variable set and unset is the A/B.
- HIP and SYCL are unverified beyond the fact that the file has no per-backend code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.