Pull requests / #2833

#2833 prefill: STRATA_PREFILL_KEEP (128) is the slots a loan leaves the cache

open · @gopinath87607 · 0 comments · View on GitHub

Server & APIAMD / HIPNVIDIA / CUDA

Description

## What this changes

`STRATA_PREFILL_KEEP=N` (**128 by default**): the expert-cache slots a loan must leave resident while a prompt chunk
borrows the cache's tail. On a small cache it is this number, not the auto lend's 90% cap, that picks the chunk. On
the Q8_0 four-way's 691- and 937-slot stages it leaves 563 and 809 slots where the cap would leave 621 and 843.
`STRATA_GLM_PREFILL_KEEP` is accepted as the same knob.

**The clamp is up, not down**: a lend has to leave the elastic K/V a region to grow into (its floor is
`STRATA_KV_GROW_FLOOR`, 128 by default), so a smaller `N` is clamped to that floor and the start says so — lower both
together to lend more of the cache and let the chunk grow at the K/V's expense. A larger `N` keeps more experts
resident for the prompt's own tokens and buys a smaller chunk.

The number is the same one the auto scan, the per-stage lend regions, the serve loan, the smallest cache a start
accepts and the vram-elastic reserve used before; this change routes all of them through one reader of the variable.

From the Maya checkout (`project-maya`), local commit `71c1ca7` (2026-10-15).

## What it leaves alone

Unset, every use is 128, exactly as before — this is by construction, not a measurement, and this PR's default run
is byte-identical on that path. With the variable set to 128 (its default) the numbers are also unchanged.

## Measurements

That commit lowered its pool's floor from 13 to 11 slots a layer (`STRATA_GLM_RESERVE_MB=400`) on a four-way split and
measured, on a 99,758-token conversation, the binding part reach an 8192-token chunk: a 49,152-token piece
61,762 → 57,507 ms (-6.9%) and a whole five-piece read 161,291 → 158,938 ms, on the live split (5060/5060/3060/3060)
— where it was not a free win either (a 16,384-token piece that went from three chunks to two lost 3%,
28,716 → 29,651 ms, since the parts run a chunk as a chain and there is less left to overlap). **No number is claimed
for our rig**: the four-way Q8_0 runs so far used 128, which is this default.

## Checks

- **Default path:** unset = 128 everywhere by construction.
- **Build:** `src/program/generate.cpp` compiles clean against this branch's headers with the build's own flags
  (`-fsyntax-only`; one pre-existing unused-variable warning in unrelated code). The development branch that also
  carries the change built fully with ninja (CUDA 13.2).
- **HIP / SYCL:** not compiled here (no toolchain on the machine). `generate.cpp` carries no per-backend code, so both
  builds compile the same change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.