#6273 [Engine] 16 GB cards: guard the ~1 GB VRAM floor during KV growth and document the per-format context limit

closed · @xvcezrqpbxtpzblrt · 0 comments · View on GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindows

Description

### Area
[Engine]

### Desktop app, Terminal or source?
Desktop app / engine — measured on the released Windows engine 0.1.62 with `--kv-grow`.

### Why this matters
16 GB VRAM + 64 GB RAM is the single most common laptop/desktop configuration. Long context is what lets a user run long tasks (long agent runs, long documents, long conversations), so how large a context a 16 GB card can actually sustain largely decides Strata's value for ordinary users. This issue reports the measured per-format context ceiling for two models, with prefill/decode speeds, and asks for changes that are within the author's control.

### Measured setup
- RTX 3080 Ti Laptop, 16 GB VRAM, 64 GB RAM, Windows 11.
- engine 0.1.62, vision + draft head on.
- Model A — IQ3_S (Qwen3.8-Flash-Next IQ3_S): `--kv-grow`, `--vram-reserve-mib 1216`.
- Model B — swift-IQ2_XS (Swift-Qwen3.8-Flash-Next IQ2_XS): native config with `--kv-resident 32768` (KV streamed to RAM; `--kv-grow` effectively off). See the dedicated section below.
- For each KV format and context size, a single prompt at ~80% of `--max-context` (the default compaction point) was sent; `nvidia-smi` sampled free VRAM every 3 s throughout the request. Each point has 62–297 samples, minimum occurring near the end of prefill. Floor criterion: **min free VRAM >= 1024 MiB**.
- Prefill speed is measured over the full hundreds-of-thousands-token prompt and is reliable. Decode speed is measured over only 24 generated tokens, so treat it as a rough trend indicator, not a precise value.

### Measured result 1 — Model A (IQ3_S): at full 512K, only q4_0 keeps the floor
| KV format | Full-context KV size | Reached 419k compaction point? | Min free VRAM | Prefill | Decode |
|---|---|---|---|---|---|
| `--kv int8` | 6.70 GiB | No — `the K/V cannot grow to this prompt: no VRAM is left` | n/a (fails first) | n/a | n/a |
| `--kv k8v4` | 5.30 GiB | Yes, but the driver runs out of VRAM (engine: `could not map 120 chunks (the driver has no VRAM left)`) | 1214.4 tok/s | 36.5 tok/s |
| `--kv q4_0` | 3.66 GiB | Yes | 1364 MiB | 1193.2 tok/s | 31.7 tok/s |

At 419k tokens both k8v4 and q4_0 report `the cache being at its floor` — the expert cache has given all it can and the K/V starts taking fresh VRAM. That is when free VRAM collapses. Speeds in this table are from a dedicated full-512K run (prompt 419068 tokens); the step-up runs in the next table used a different prompt length (e.g. 419498 for q4_0), so a format's speed differs slightly between the two tables. Both are measured, not estimated.

### Measured result 2 — Model A (IQ3_S): maximum context per format that keeps ~1 GB free
The context was stepped up for each format until the 1 GB floor was breached. All numbers below are freshly measured, not estimated:

| KV format | `--max-context` | 80% compaction point (measured prompt tokens) | Min free VRAM | Prefill | Decode | Verdict |
|---|---|---|---|---|---|---|
| `--kv int8` | 327680 (320K) | 262212 | 1214 MiB | 1402.7 tok/s | 35.0 tok/s | keeps 1 GB (190 MiB margin) |
| `--kv k8v4` | 393216 (384K) | 314641 | 1350 MiB | 1255.2 tok/s | 43.6 tok/s | keeps 1 GB (326 MiB margin) |
| `--kv q4_0` | 524288 (512K) | 419498 | 1364 MiB | 1239.0 tok/s | 30.3 tok/s | keeps 1 GB (340 MiB margin) |

Full step data (adjacent points show where the floor breaks):
| Format | max-context | compaction tokens | min free VRAM | Prefill | Decode | Verdict |
|---|---|---|---|---|---|---|
| q4_0 | 512K | 419498 | 1364 MiB | 1239.0 tok/s | 30.3 tok/s | keeps |
| k8v4 | 384K | 314641 | 1350 MiB | 1255.2 tok/s | 43.6 tok/s | keeps |
| k8v4 | 416K | 340855 | 1028 MiB | 1241.7 tok/s | 45.0 tok/s | borderline (4 MiB above floor) |
| k8v4 | 448K | 367070 | 726 MiB | 1209.0 tok/s | 37.0 tok/s | breaks floor by 298 MiB |
| int8 | 320K | 262212 | 1214 MiB | 1402.7 tok/s | 35.0 tok/s | keeps |
| int8 | 336K | 275319 | 877 MiB | 1377.3 tok/s | 36.7 tok/s | breaks floor by 147 MiB |
| int8 | 384K | 314641 | 466 MiB | 1349.7 tok/s | 37.2 tok/s | breaks floor by 558 MiB |

So on a 16 GB card the largest context that keeps ~1 GB free is roughly: **int8 ~320K, k8v4 ~384K, q4_0 512K**. Beyond that, higher precision means a larger K/V per token and the floor is broken.

Note on the 416K k8v4 point: it holds 1028 MiB free, only 4 MiB above the 1024 MiB floor, and it has the best decode speed of any IQ3_S point that still holds the floor (45.0 tok/s). Its prefill (1241.7 tok/s) is slightly slower than the shorter safe points (384K k8v4 1255.2, 320K int8 1402.7), so the tradeoff is context length plus decode speed, not raw prefill speed. A user may reasonably accept the 4 MiB overshoot for the extra context and higher decode speed — but the UI currently offers no way to enter this value, and there is no margin for even small variation. Both points matter: expose custom context numbers, and let the user decide how much VRAM to reserve.

### Measured result 3 — Model B (swift-IQ2_XS): 1M context, floor never the bottleneck
This model's native config uses `--kv-resident 32768`: 32768 KV cells stay in VRAM and the rest streams to RAM, so `--kv-grow` is effectively off. All three KV formats reach 1M (1048576) while keeping ~1.7–1.8 GiB free — the 1 GB floor is not the limiting factor here. All numbers are long-prompt measurements.

| KV format | max-context | 80% compaction point (measured prompt tokens) | Min free VRAM | Prefill | Decode | Verdict |
|---|---|---|---|---|---|---|
| `--kv int8` | 1048576 (1M) | 838928 | 1782 MiB (296 samples) | 931.5 tok/s | 42.0 tok/s | keeps 1 GB (758 MiB margin) |
| `--kv k8v4` | 1048576 (1M) | 838928 | 1774 MiB (297 samples) | 929.2 tok/s | 53.2 tok/s | keeps 1 GB (750 MiB margin) |
| `--kv q4_0` | 1048576 (1M) | 838928 | 1768 MiB (285 samples) | 968.3 tok/s | 49.9 tok/s | keeps 1 GB (744 MiB margin) |

Additional points (min free VRAM):
| max-context | int8 | k8v4 | q4_0 |
|---|---|---|---|
| 384K | 1671 MiB | not measured | not measured |
| 512K | 1774 MiB | 1774 MiB | 1770 MiB |
| 640K | 1776 MiB | not measured | not measured |
| 768K | 1776 MiB | 1774 MiB | 1770 MiB |
| 1024K | 1782 MiB | 1774 MiB | 1768 MiB |
| 2048K | not measured | not measured | short-prompt startup only, no long-prompt run — not recommended |

Important: swift reaches 1M because KV streams to RAM, not because the weights are small. If `--kv-resident` is removed and KV is forced to grow fully in VRAM (the IQ3_S style), swift int8 is rejected outright at 384K with `the K/V cannot grow to this prompt: no VRAM is left` — its expert cache takes 6017 slots (IQ3_S ~3621), leaving no room for KV growth. Caveats at 1M: int8 hits an expert-memory pinning failure and the engine falls back to working-set (correctness unaffected, but a memory-pressure signal); larger contexts shrink the expert cache and hurt performance.

The two models are measured under fundamentally different KV strategies and should not be compared directly. IQ3_S keeps KV fully in VRAM and is floored at ~320K–512K; swift streams KV to RAM and reaches 1M at the cost of KV not residing in VRAM.

### Problem 1 — the VRAM floor is not guarded
`--vram-reserve-mib` only reserves memory at startup. Once the K/V grows, it maps chunks until the driver has nothing left: at full 512K, k8v4's engine log reports `the elastic K/V could not map 120 chunks (the driver has no VRAM left)` — well below the ~1 GB region where WDDM starts paging GPU memory to system RAM. A 1 GB floor is non-negotiable for these users: when Strata drains it, the machine can still run Strata but nothing else — the browser and other tools break. There is no knob that says "never let K/V growth take the last 1 GB", and no signal when a chosen context is too large for the KV format on that card.

### Problem 2 — exceeding the safe context is silent until it is too late
There is no warning when a configured context is larger than the format's safe ceiling on that card. k8v4 at 448K silently breaks the 1 GB floor (726 MiB) but the request still completes; int8 at 336K/384K breaks the floor and at 512K fails outright with HTTP 400 `the K/V cannot grow to this prompt: no VRAM is left`; its engine log also shows `the elastic K/V could not map 100 chunks (the driver has no VRAM left)`. The user gets no indication at setup time that the chosen context will not hold the floor.

The opposite failure is just as misleading. The model page's context-fit warning (`NativeModels.tsx`, `contextIsTight` / `contextCacheGb` in `apps/host/src/lib/native-fit.ts`) estimates the whole conversation cache as if it all stayed in VRAM — 13 layers x the per-token KV bytes of the chosen format, 5.4% factor, plus a fixed 121 MiB. At 1M with int8 that is about 14.3 GB, so the page tells a 16 GB user the context "takes about 14.2 GB of graphics memory and leaves little room for the model". That number is wrong for any run with `--kv-resident`: with the default 32K resident window the KV streams to system RAM and only the window stays in VRAM. Model B below ran a 1M context on this 16 GB card with 1,782 MiB still free — the setup page would have told the user it could not fit. The warning counts the entire KV as VRAM and ignores streaming, so it scares users away from a context their machine actually runs, while the same page gives no warning at all in the cases above where the floor really is broken.

### Problem 3 — no way to enter an arbitrary context number, and no way to tune the VRAM reserve
The UI only offers preset context sizes. The measured sweet spots are often between presets — e.g. 416K (425984 tokens) for IQ3_S k8v4, the largest k8v4 context that still holds ~1 GB — but a user cannot type that number. Likewise `--vram-reserve-mib` is fixed; not every user wants to reserve the same amount. Reserving too much wastes VRAM for users who do not need a 1 GB floor; others want a strict floor. Neither value is user-configurable today.

### Proposed changes (behavioral, not physical)
1. **Guard a configurable VRAM floor during K/V growth.** Add a setting (reuse `--vram-reserve-mib` or a new `--kv-vram-floor-mib`) that stops K/V from mapping further chunks once free VRAM approaches the floor (e.g. 1 GB), instead of draining until the driver reports no VRAM left. Keep the K/V at its current length and let the client compact.
2. **Warn when the configured context exceeds the format's safe ceiling.** At session start (or per request), estimate the K/V growth for the chosen KV format and context and, if it would breach the floor on that card, emit an explicit warning before the run — e.g. "int8 at 512K cannot keep 1 GB free on this 16 GB card; use <= 320K or a smaller KV format". Also emit a clear signal at the floor ("K/V reached the VRAM floor at ~N tokens — compact now") so the user can compact before an abrupt 400. Same class of fix as #6268 (the startup error should suggest `--kv-grow`). Make the setup page's context-fit warning use the same streaming-aware estimate: when `--kv-resident` is set, count only the resident window in VRAM and the rest in RAM, so it stops reporting ~14 GB for a 1M context that actually runs.
3. **Document a per-format, per-VRAM safe-context table.** Ship the measured ceiling (int8 ~320K, k8v4 ~384K, q4_0 512K on 16 GB for IQ3_S; 1M for swift with KV resident) in the docs so users can pick the largest context that still leaves the machine usable. Include measured prefill/decode speeds. Long context is the feature users want most; telling them the safe ceiling is more useful than letting them hit a 400 or silently break the floor.
4. **Let users enter an arbitrary context size.** Add a numeric input to the context setting so users can enter any token count or K value (e.g. 416K = 425984), not just preset list values. Validate a sensible range and pass the value through to `--max-context`. This lets users pick their own point on the speed/VRAM tradeoff instead of being stuck with presets.
5. **Let users configure the VRAM reserve.** Expose `--vram-reserve-mib` as a user setting in MiB.

Related on this site

Quick links to related pages.