#6657 [Engine] Single-GPU start with a too-large K/V cache exits instead of falling back to KV streaming
open · @xvcezrqpbxtpzblrt · 0 commentaires · Sur GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants
Description
Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.
**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.
> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Source (the CUDA/HIP engine build, `engine/src/program/generate.cpp`)
### What happened
On the CUDA/HIP build — the build virtually every user runs — a start whose K/V cache does not fit in VRAM is **all-or-nothing**: the engine never tries to shrink the K/V working set. `o.kv_resident` is set only from the command line (default `0`, i.e. the *entire* `--max-context` resident in VRAM), and the session is allocated once at that full size. If the allocation fails, the process prints a bare `session state allocation failed` and exits 1. There is no automatic fallback to K/V streaming to pinned RAM, no retry with a smaller resident window, and no actionable numbers in the error.
The only KV-window-reduction logic in this build is inside the **multi-GPU layer-split auto search** (it runs only when `multi_gpu && split_auto` and no placement was found). A single-GPU start — by far the common case — can never reach it.
The project's **own SYCL port already implements exactly the missing mechanism**: it auto-selects the largest KV window that fits free VRAM before allocating the session, checks that pinned RAM can hold the streamed copy, and prints what it chose. That code has no counterpart in `src/program/generate.cpp`. This is a missing protection in the primary build, not a design trade-off.
Because the K/V cache is the largest VRAM consumer and grows linearly with context, any source of VRAM pressure (a longer context, `--batch` slots, a larger dense set, another app holding VRAM, driver-reserved memory) turns into a hard start failure instead of a graceful slowdown. Conversely, users who do happen to know about `--kv-resident` must pick the window by hand, with no guidance on what fits. A general, automatic "VRAM overflow → KV streaming" path would make starts robust on every machine, for every model and quant, and for every context size where the whole context does not fit; where the whole context does fit, the correct behaviour (no streaming) is unchanged.
### Smallest steps or command that reproduce it
Static analysis only; the following is the code path, not a measured run.
1. Build the CUDA (or HIP) `strata generate` target from the official source.
2. Start with a single GPU and a context whose K/V does not fit the free VRAM, e.g. `--max-context 131072`, without passing `--kv-resident`.
3. `o.kv_resident` stays at its default `0`; `qsa_set_kv_resident(0)` is applied; the session is allocated at full-context size; the allocation fails and the process exits with `strata generate: session state allocation failed`.
4. Expected: the engine should behave like the SYCL build — auto-select a resident window, stream the rest from pinned RAM, keep the requested context, and say so.
### Reproduction status
Unconfirmed static analysis. No binary was built or run; all findings below are from reading the official source only.
### Strata version
0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)
### Graphics card, memory and system
Not measured (static analysis). The finding is backend-specific (CUDA/HIP `generate` vs. the SYCL port), not tied to a particular GPU: it applies whenever the requested K/V exceeds free VRAM.
### Model, size and settings
Not model- or quant-specific. Relevant knob: `--max-context` (default `4096`) and `--kv-resident` (default `0`, whole context resident).
### Logs
**Source evidence (verbatim, file:line).** Paths are relative to the repository root.
1. `kv_resident` defaults to `0` (whole context in VRAM) and is only ever set from the CLI:
`engine/src/program/generate.cpp:819`
```cpp
int64_t kv_resident = 0; ///< KV streaming: resident cells per QSA layer (0: all in VRAM)
```
`engine/src/program/generate.cpp:3835`
```cpp
else if (a == "--kv-resident") o.kv_resident = std::atoll(next("--kv-resident"));
```
`engine/src/program/generate.cpp:4897-4901`
```cpp
if (o.kv_resident < 0) {
std::fprintf(stderr, "strata generate: --kv-resident must be >= 0\n");
return 2;
}
strata::core::qsa_set_kv_resident(o.kv_resident);
```
2. The session is then allocated once at full-context size; on failure there is no smaller-window retry and no guidance:
`engine/src/program/generate.cpp:7497-7500`
```cpp
if (cudaMalloc(&sbuf, strata::core::session_bytes(g, o.max_context, K, 0, hi0)) != cudaSuccess) {
std::fprintf(stderr, "strata generate: session state allocation failed\n");
return 1;
}
```
3. A complete search over `qsa_set_kv_resident` call sites in the CUDA/HIP build shows the only other writes are in the multi-GPU layer-split spill walk, which is gated on a failed split search and `spill_discrete_gpu()`:
`engine/src/program/generate.cpp:7277-7284`
```cpp
// THE LAYER SPLIT'S SPILL WALK, STEP 2: THE KV WINDOW DOWN. No placement can start the prompt path with the window asked for: the sessions of the cards are smaller with a
// smaller one (the rest of the K/V streams from pinned RAM), so the search runs again at 20480, 8192, 4096 cells a layer (the engine's smallest) and keeps the first window
// that has a startable placement. The context and the number of requests are not touched. Never reached by a start that has a placement.
if (best.empty() && std::getenv("STRATA_SPILL_OFF") == nullptr && spill_discrete_gpu()) {
const int64_t kv_asked = o.kv_resident;
for (int64_t w; (w = strata::program::split_walk::next_kv_window(o.kv_resident, o.max_context, strata::core::qsa_kv_resident_min())) > 0;) {
o.kv_resident = w;
strata::core::qsa_set_kv_resident(w);
```
The enclosing split search itself is `multi_gpu && split_auto` (`engine/src/program/generate.cpp:5678-5680`), so a single-GPU start cannot reach this fallback. For comparison, the pinned-KV-allocation failure path is also terse (`engine/src/core/layer.cpp:927-931`):
```cpp
if (p.mode == 1) std::fprintf(stderr, "strata: KV streaming: cannot pin %.2f GiB of RAM for a layer's KV copy "
"(%.2f GiB pinned so far): RAM or the driver refused to pin it - lower the context, "
"or run without --kv-resident\n", (double) bytes / 1073741824.0,
(double) g_kv_host_bytes / 1073741824.0);
```
4. The SYCL port has the protection that the CUDA/HIP build lacks — auto-select a fitting window before the session allocation, refuse streaming if pinned RAM is short, and report the decision:
`engine/sycl/src/program/generate.cpp:4188-4192`
```cpp
// A context the sessions cannot hold in VRAM is not a reason to stop (owner rule: a long context and requests at once never shrink the context, they run
// slower): with no --kv-resident given, the K/V streams from RAM, the resident window as large as the card leaves room for.
// Needed: every session (the main one and each request slot), what loads after them (the draft layer, the native head, the verify buffers), the reserve,
// the prompt path's smallest own buffers (chunk 256) and a cache of 128 slots.
if (o.kv_resident <= 0 && o.max_context > 20480 && !multi_gpu && !o.kv_grow && o.expert_cache != 0) {
```
`engine/sycl/src/program/generate.cpp:4199-4204`
```cpp
free_k = strata::core::device_free_bytes(); (void) total_k;
auto need_at = [&](int64_t res) -> int64_t {
strata::core::qsa_set_kv_resident(res);
return n_sess * (int64_t) strata::core::session_bytes(g, o.max_context, K, 0, g.n_layers) + fixed_b;
};
int64_t pick = strata::core::pick_kv_resident(o.max_context, (int64_t) free_k, need_at);
```
`engine/sycl/src/program/generate.cpp:4216-4228`
```cpp
const int64_t host_need = n_sess * (int64_t) o.max_context * 13 * 1056 + (6ll << 30);
if (host_avail >= 0 && host_avail < host_need) {
std::fprintf(stderr, "strata generate: the context does not fit in VRAM and %.1f GiB of free RAM is less than the %.1f GiB KV streaming would take: not turned on\n",
(double) host_avail / 1073741824.0, (double) host_need / 1073741824.0);
pick = 0;
} else {
std::fprintf(stderr, "strata generate: %lld tokens of context in %lld sessions do not fit in the %.2f GiB free: KV streaming on, %lld cells a layer in VRAM "
"(--kv-resident %lld; the rest of the K/V in RAM, so decode and prompts run slower)\n",
(long long) o.max_context, (long long) n_sess, (double) free_k / 1073741824.0, (long long) pick, (long long) pick);
}
}
o.kv_resident = pick;
strata::core::qsa_set_kv_resident(o.kv_resident);
```
`engine/sycl/include/strata/core/short_cache.hpp:42-50`
```cpp
/// `need_at(window)` is the bytes the sessions and everything after them need with that KV window (0: the whole context in VRAM); `free_b` what the card has.
/// Returns 0 when the whole context fits (no streaming), else the largest window that fits, else the smallest the engine takes (20480).
template <class NeedAt>
inline int64_t pick_kv_resident(int64_t ctx, int64_t free_b, NeedAt need_at) {
if (need_at((int64_t) 0) <= free_b) return 0;
for (const int64_t w : {(int64_t) 65536, (int64_t) 32768, (int64_t) 20480})
if (w < ctx && need_at(w) <= free_b) return w;
return 20480;
}
```
### Root cause
The single-GPU CUDA/HIP startup sequence is:
1. `--kv-resident` is parsed into `o.kv_resident` with default `0` (`generate.cpp:819, 3835`).
2. `qsa_set_kv_resident(o.kv_resident)` publishes that value (`generate.cpp:4901`).
3. The one and only session allocation calls `cudaMalloc(session_bytes(g, o.max_context, ...))` (`generate.cpp:7497`). With `kv_resident == 0`, `kv_plan` keeps every page in VRAM (`engine/src/core/layer.cpp:738-755`), so `session_bytes` scales with the whole context and the allocation is large.
4. On failure the code exits (`generate.cpp:7498-7499`). The only KV-window shrinkage in this build lives behind the multi-GPU split search (`generate.cpp:5678-5680, 7277-7296`), which a single-GPU start cannot enter.
So there is no code that, on VRAM shortfall, lowers the resident window and retries, and no code that measures whether pinned RAM could hold the streamed copy. The SYCL build's `pick_kv_resident` block (`sycl/src/program/generate.cpp:4188-4228`) is the missing implementation on the primary build.
### Proposed fix
1. Port the SYCL `pick_kv_resident` logic (`engine/sycl/include/strata/core/short_cache.hpp:42-50`) and the pre-allocation auto-selection block (`engine/sycl/src/program/generate.cpp:4192-4228`) into `engine/src/program/generate.cpp`, inserted before the session `cudaMalloc` at line 7497. Keep an explicit `--kv-resident` as an override; only auto-select when it was not given.
2. Keep the pinned-RAM admission check and the explicit report (`KV streaming on, N cells a layer in VRAM ...`), so a context is never silently streamed into pinned memory that cannot hold it.
3. When no window fits or pinned RAM is insufficient, make the failure actionable using the same measured numbers instead of the bare `session state allocation failed` (`generate.cpp:7498`), including the recommended `--kv-resident` / smaller-context values.
4. Acceptance check (static criteria): with a single GPU and `--max-coSur le site
Liens install, modèles, releases.