Pull requests / #3309
#3309 [Engine] warm up the prompt path once before READY
open · @gopinath87607 · 0 comentários · No GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
Descrição
> **Why this PR was recreated.** The original (#2832) was opened from the `gopinath87607/Strata` fork. Deleting that fork deleted its branches, and GitHub closes every pull request whose branch lived in a deleted fork - a closed PR whose source branch is gone cannot be reopened. This is the same work from a new fork, rebased onto current `main` (`e88ee39d`); the conflict with upstream's `chunk_vram` prompt planner in `generate.cpp` was resolved. It replaces the closed #2832, which was closed by the fork's deletion, not by a maintainer. ## Problem and result A session's first prompt pays what no later prompt pays: the first chunk carries every kernel's first launch, the prompt path's own allocations and the tiers' first reads. After this change, `STRATA_WARM_PROMPT=N` (off by default) reads one synthetic N-token prompt through the whole part chain **before the engine says READY**, so the first real request does not pay it. Unset or `0`, the block is skipped and the engine says READY exactly as before. ## Related discussion The original pull request and its discussion: #2832 (closed when its fork was deleted; this PR replaces it). The work comes from the Maya checkout (`project-maya`), local commit `cc58309` (2026-10-14). ## Approach `STRATA_WARM_PROMPT=N` (`--serve`): one synthetic prompt of N tokens (32 if you ask for less) is read through the whole part chain before READY. `STRATA_GLM_WARM_PROMPT` is accepted as the same knob. The synthetic ids are spread over the vocabulary (`100 + (i * 131) % 20000`), so the routes are not one expert's. It takes the same loan a request takes (the slots the prompt path is laid out in go non-resident and are refilled from the arena afterwards), and every participant's state is zeroed before and after it, so no conversation can see it. A prompt path that cannot run prints why and the first prompt pays it; the start never fails because of it. Unset or `0` (the default), the whole block is skipped and the engine says READY exactly as before. Nothing outside the block changes; this is by construction. ## Validation Tested revision: `af867e73`, one commit on current `main` (`e88ee39d`). Environment: Linux, CUDA 13.2 (V13.2.86), `sm_86` + `sm_120`; the build machine is the Q8_0 4-way rig (2x RTX 3060 12 GB, 2x RTX 5060 8 GB). **Passed:** - `git rebase e88ee39d` - the conflict in `generate.cpp` with upstream's `chunk_vram` prompt planner was resolved; the branch's diff is this feature only. - `ninja -C build strata` at `af867e73` - exit 0 (CUDA 13.2, `sm_86` + `sm_120`). - `-fsyntax-only` on `src/program/generate.cpp` with the build's own flags - clean; one pre-existing unused-variable warning in unrelated code. This was run before the rebase; the full build above covers the rebased head. - Reported by blange48 on 4x RTX 5080 16 GB (2026-10-17; not run by us; Linux, layer split 12,24,36, Qwen3.8-Flash-Next IQ3_S, `--batch 8 --batch-groups 4`, v0.1.45 + this PR + #2673, one run per case, warm page cache): time to READY 102 s unset vs 101 s at `STRATA_WARM_PROMPT=512`; first request (a ~325-token prompt) time to first token 1.00 s unset vs 0.72 s at 512; later requests were unaffected. - Maya checkout's issue #8 reported 95-133 ms/token on a first prompt against 23-44 warm - that checkout's number on its rig. **Failed:** - none observed. **Not run:** - Our own hardware: the reported run above is not ours. - More than one run per case, and a start with a cold page cache. - Serving the rebased head: only the build was re-run after the rebase; no GPU run was made at `af867e73`. - HIP and SYCL builds: no toolchain on this machine. `generate.cpp` carries no per-backend code, so both builds compile the same change. ## Limits - The report is one run per case, so its 0.28 s first-token gain is not a distribution. - The reported build also carried #2673, whose different-greedy-answers issue is being looked at there; this PR was off in those comparisons, so it does not look involved. - On this engine the knob is how you measure it: the first request's time to first token, or its `PP` line, with the variable set and unset is the A/B. - HIP and SYCL are unverified beyond the fact that the file has no per-backend code. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.