Issues / #2310

#2310 [Feature]: --kv-grow with --prefill auto: let the prompt path's loan shrink as the K/V grows (auto stops a 262K config at ~115K on a 12 GB card)

open · @JawadN00r · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux

Description

**Goal:** with `--kv-grow`, get both `--prefill auto`'s fast prompts while a conversation is short *and* the full
`--max-context` when it gets long, without choosing one at startup.

**What happens today.** `--prefill auto` sizes the prompt path's loan once at startup (the largest chunk up to 8192
whose buffers fit in ≤85% of the expert cache; `plan_lend` in `generate.cpp`), and `--kv-grow` grows the K/V only into
the slots *below* that loan (`kvg_start(lend_first)`). Neither looks at `--max-context`. On a 12 GB card the loan is so
large that the K/V cannot reach the configured context:

```
strata serve: prompt chunk auto: 8192 tokens, a 384-slot ring
strata serve: the prompt path borrows 2456 CUDA0 cache slots (3.99 GiB)
strata generate: elastic K/V: 16384 of 262144 cells in VRAM (0.25 of 3.35 GiB); the expert cache (3258 slots) gives the K/V room below slot 802 as the context grows
...
strata: K/V grown to 98304 cells (1.27 GiB); ... (2584 of 3258 slots hold experts) - and new VRAM, the cache being at its floor
strata: the elastic K/V could not map 2 chunks (the driver has no VRAM left)
strata: the K/V could not grow to 122880 cells (no error)
```

The client (an agent harness, told the model has 262,144 tokens) then gets an error at ~100-115K and the task is lost.
Nothing at startup says that this loan and `--max-context 262144` cannot both be served.

**Setup:** v0.1.43.1 (`a0116b2`), Linux, RTX 3060 12 GB (driver 595.58.03, CUDA 13.0 build), Ryzen 7 5700X,
62 GiB DDR4-3200; UkisAI Swift 1.5 Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS (3258 cache slots), `--expert-cache auto
--prefill auto --spec 4 --spec-min-p 0.5 --mtp mtp/rt --max-context 262144 --kv int8 --kv-grow --prompt-cache 2
--vram-reserve-mib 512`.

**Measured (same model and flags, only the prompt chunk changed):**

| prompt chunk | loan (slots) | K/V room below slot | decode 3×512 @2K (tok/s) | prefill 750 / 7K / 18.5K (tok/s) | longest prompt |
|---|---|---|---|---|---|
| `auto` (8192) | 2456 | 802 | 65.4 / 63.2 / 63.8 | 519 / 978 / 1012 | **stops at ~115K** |
| `4096` | 1538 | 1720 | 65.5 / 67.5 / 65.1 | 527 / 907 / 946 | stops at ~245K |
| `3072` | 1312 | 1946 | 64.8 / 64.2 / 67.6 | 524 / 867 / 900 | 251K ok, ~30 slots short of 262K |
| `2048` | 1078 | 2180 | 64.8 / 66.1 / 64.3 | 524 / 813 / 822 | **261,431 tokens ok: K/V 262144 cells, prefill 726, decode 42.2** |
| `auto` + `STRATA_PREFILL_LEND_PCT=33` (picks 2816, 227-slot ring) | 1042 | 2216 | 61.5 / 65.9 / 68.0 | 533 / 744 / 758 | 251K ok |
| `auto` + `STRATA_PREFILL_LEND_PCT=38` (picks 3584, 227-slot ring) | 1214 | 2044 | 68.4 / 67.9 / 70.3 | 522 / 763 / 838 | 251K read, but "could not grow to 253952 cells" on the way |

(One 256-token answer per long prompt: decode at 251K was 42.8-61.5 tok/s across the arms, so the K/V taking slots
from the cache costs little at this length.)

**Workaround today:** a fixed `--prefill 2048`. It reaches the full 262K, but every prompt is read 17-19% slower at
7K-18.5K tokens than with `auto`, also in the short conversations that never need the room. `STRATA_PREFILL_LEND_PCT`
works as a cap, but the smaller ring it picks makes prompts slower still (744-838 tok/s), and it is not documented.

**What would help (any of these, in order of value):**
1. **A loan that steps down when the K/V needs the room.** When `kvg_ensure` reaches the cache floor below the loan,
   re-plan the prompt path one step smaller (8192 → 4096 → 2048 ...; the `relend` path used under `--vram-elastic`
   looks like the place) and hand the freed slots to the K/V; step back up when a short request trims the K/V. Short
   conversations keep 8192-token chunks; a long one pays the smaller chunk only while it is long.
2. **`auto` aware of `--kv-grow` and `--max-context`:** pick the largest chunk whose loan still leaves the K/V room for
   `--max-context` cells (here: ≤ ~1300 of 3258 slots), or accept `--prefill auto:max-context` to ask for that.
3. **At least a startup line** when the two cannot both be served, e.g. "with this loan the K/V can grow to ~98K of
   --max-context 262144 cells; --prefill 2048 reaches it", like the existing VRAM-reserve warning.

Happy to test a branch on this card.

Sur le site

Liens install, modèles, releases.