Pull requests / #2833
#2833 prefill: STRATA_PREFILL_KEEP (128) is the slots a loan leaves the cache
open · @gopinath87607 · 0 commentaires · Sur GitHub
Server & APIAMD / HIPNVIDIA / CUDA
Description
## What this changes `STRATA_PREFILL_KEEP=N` (**128 by default**): the expert-cache slots a loan must leave resident while a prompt chunk borrows the cache's tail. On a small cache it is this number, not the auto lend's 90% cap, that picks the chunk. On the Q8_0 four-way's 691- and 937-slot stages it leaves 563 and 809 slots where the cap would leave 621 and 843. `STRATA_GLM_PREFILL_KEEP` is accepted as the same knob. **The clamp is up, not down**: a lend has to leave the elastic K/V a region to grow into (its floor is `STRATA_KV_GROW_FLOOR`, 128 by default), so a smaller `N` is clamped to that floor and the start says so — lower both together to lend more of the cache and let the chunk grow at the K/V's expense. A larger `N` keeps more experts resident for the prompt's own tokens and buys a smaller chunk. The number is the same one the auto scan, the per-stage lend regions, the serve loan, the smallest cache a start accepts and the vram-elastic reserve used before; this change routes all of them through one reader of the variable. From the Maya checkout (`project-maya`), local commit `71c1ca7` (2026-10-15). ## What it leaves alone Unset, every use is 128, exactly as before — this is by construction, not a measurement, and this PR's default run is byte-identical on that path. With the variable set to 128 (its default) the numbers are also unchanged. ## Measurements That commit lowered its pool's floor from 13 to 11 slots a layer (`STRATA_GLM_RESERVE_MB=400`) on a four-way split and measured, on a 99,758-token conversation, the binding part reach an 8192-token chunk: a 49,152-token piece 61,762 → 57,507 ms (-6.9%) and a whole five-piece read 161,291 → 158,938 ms, on the live split (5060/5060/3060/3060) — where it was not a free win either (a 16,384-token piece that went from three chunks to two lost 3%, 28,716 → 29,651 ms, since the parts run a chunk as a chain and there is less left to overlap). **No number is claimed for our rig**: the four-way Q8_0 runs so far used 128, which is this default. ## Checks - **Default path:** unset = 128 everywhere by construction. - **Build:** `src/program/generate.cpp` compiles clean against this branch's headers with the build's own flags (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also carries the change built fully with ninja (CUDA 13.2). - **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so both builds compile the same change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.