#6268 [Engine] The no-VRAM-left-for-expert-cache error never suggests --kv-grow, only quality-reducing fallbacks
open · @xvcezrqpbxtpzblrt · 0 comentarios · En GitHub
BenchmarksNVIDIA / CUDAModels & quantsDocumentationWindows
Descripción
### Area [Engine] ### Desktop app, Terminal or source? Source (build) — the engine's own startup error text in `src/program/generate.cpp`. ### What happened On a 16 GB card, starting a 512K-context session with INT8 KV fails at startup. The engine's own error is: ``` strata generate: expert cache auto: 1.92 GiB free, 2048 MiB reserved (+184 MiB for the draft head) -> 0 slots strata generate: no VRAM is left for the expert cache: it needs at least 332 slots (737 MiB), about 1000 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 7361 MiB at 524288 tokens), --kv q4_0, the English draft subset ..., images on the CPU, or close other programs that use the GPU ``` Every remedy this message offers takes something away from the user: a shorter context, a lower-precision KV (`q4_0`), a smaller draft, no images, a smaller reserve. It never mentions `--kv-grow` — the one built-in switch that addresses the actual cause **without** giving up speed or precision — and in the current source (commit `13eedf55`) the message still does not mention it. ### Smallest steps or command that reproduce it On an RTX 3080 Ti Laptop 16 GB, start a 512K-context (524288-token) session with INT8 KV and pictures on. The start fails with the message above. (Original failing log: Strata 0.2.2 / engine 0.1.47.) ### Expected The "to make room" hint should also name `--kv-grow`: the elastic-KV switch that keeps the whole K/V on the GPU (no streaming, no PCIe KV reads) but maps physical VRAM only for the cells actually reached, returning the expert-cache slots the message says are missing. It is the only lever, listed or not, that does not trade away context length, speed or KV precision. ### Actual The hint lists only quality/speed-reducing fallbacks. A user following it ends up lowering precision or context even though `--kv-grow` exists for exactly this failure. It is also off by default (`generate.cpp:673`) and gated on a non-obvious set of preconditions (`:6774-6789`), so it is easy to miss even for someone who has heard of it. ### Root cause, and what it is not (source evidence, commit 13eedf55) - `src/program/generate.cpp:6791` allocates `session_bytes(g, o.max_context, ...)` in one `cudaMalloc` — the KV is sized for the full `--max-context` before the expert cache is sized at `:7949`. That is the 7361 MiB the error itself reports; `:8190-8204` recomputes `session_mib` from that same call. - This is **not** a case of prompt-path loans not being returned. Loan records `p.lent` (`:17219`) are refilled before the request ends (`:18137`, log line "prompt done (slots refilled)"), with an idempotency guard (`:17119`) and a runtime check that refuses VRAM changes while a loan is out (`:14233-14234`). The shortage is the pre-placed KV, not VRAM borrowed and forgotten. - `:673` defaults `kv_grow = false`; `:6774-6789` moves the K/V pools to a CUDA VMM range and maps only 16 384 cells initially; `src/core/layer.cpp:792` stops counting those pools in `session_bytes` when elastic. - The behaviour is already published: `docs/DETAILS.md:1443` records that at `--max-context 262144` the default engine did not start while `--kv-grow` started with 7,877 slots (94.4 tok/s), and at 196608 it improved decode 96.3 -> 98.5 tok/s. ### Related - #5876 covers the same KV-before-experts ordering, from the angle of resident-expert counts and decode speed on an 11 GB card. This issue is specifically about the startup-refusal message and its missing hint. - #5852 notes the same error message omits the KV flag that helps on their 32 GB box (`--kv-resident`, a streaming switch). Here the point is the complementary, non-streaming switch `--kv-grow`, which does not cost speed; the two switches are mutually exclusive. ### Proposed change Add one line to the message at `:8197-8204`: suggest `--kv-grow` when it is available for the build/config. Optionally, enable it by default for large contexts when its preconditions hold (single GPU, expert profile present, no resident-experts mode, VMM available, no batch) — this is a change to a default, so I am asking for scope agreement here rather than assuming it. ### Reproduction status Unconfirmed static analysis. I read the source at commit `13eedf55`; the 512K failure text is from a 0.1.47 log, and I have not re-run 512K on the current engine. A direct verification would be to start the same 512K/INT8 session with `--kv-grow` added and observe whether the engine now starts. ### Strata version Strata 0.2.2 / engine 0.1.47 (failing log); source analysed at commit 13eedf55 (2026-11-11) ### Graphics card, memory and system RTX 3080 Ti Laptop 16 GB · 64 GB RAM · Windows 11 (10.0.26300) ### Model, size and settings Swift 1.5 IQ3_XXS · 512K context (524288 tokens) · INT8 KV · pictures on ### Logs The startup error text quoted above, from the original 0.1.47 log. No keys, tokens or private logs are attached.
En el sitio
Enlaces a install, modelos, releases.