Pull requests / #2399
#2399 `--vram-elastic` with `--kv-grow`: VRAM takes its chunks where the K/V grows into (resubmits #2201)
open · @i-Vladimir-i · 0 comentarios · En GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descripción
## Title
`--vram-elastic` with `--kv-grow`: VRAM takes its chunks where the K/V grows into (resubmits #2201)
Issue: follows up #2201 (held for #2189; "send it together with the new #2189"). Builds on #2398 for HIP.
## Summary
Today the two switches exclude each other. With `--vram-elastic` the engine says `--kv-grow is off` and puts the whole
K/V in VRAM, because #533's cache is cut into segments while `--kv-grow` needs one VMM range of chunks. On an R9700
with a 262K context that costs ~1,400 cache slots, more VRAM than a game or an image encoder would borrow. This PR lets
both switches run together.
This branch is #2398 plus one commit; only the last commit is this change. It does not depend on #2398 on CUDA, and
once #2398 is in, it rebases onto main with no conflict in the code.
## What changed
`src/program/generate.cpp`:
- With both switches, the cache keeps `--kv-grow`'s arena (one range of 2 MiB chunks). `VRAM` works on the region the
K/V grows into: slots `[lo, top)`, just below the prompt path's loan.
- **Shrink:** the same eviction the K/V's growth uses (hotter experts there move into the coldest slots of the loan
region first). It is now `kvg_take`, split out of `kvg_ensure` unchanged; the chunks go to the driver instead of to
the K/V.
- **Grow:** new chunks are mapped there, and those slots are refilled with the profile's hottest experts that the GPU
does not hold. This is now `kvg_give_back`, split out of `kvg_trim` unchanged. So after a grow the cache holds what a
K/V trim would leave, not necessarily the experts the shrink took (the segments put the same ones back). The docs say
so.
- The prompt path's loan and chunk are not touched. While the cache is shrunk, the K/V keeps growing from the cache's
slots, not from the reserve.
- A small counter keeps track of the chunks given to the driver. It is settled against the arena's empty places,
because the K/V's first cells are new VRAM, and a trim below them maps those chunks into the arena too. Without that,
the count drifted by ~100 MiB and a grow reported "no VRAM" for chunks that had no place.
`docs/DETAILS.md`, `docs/VRAM_ELASTIC.md`: the new "With `--kv-grow`" section. The image-encoder section (#2196) no
longer says that `--vram-elastic` turns `--kv-grow` off.
What it leaves alone: either switch alone (`kvg_ensure` and `kvg_trim` do the same steps through the two helpers;
without `--kv-grow`, `VRAM` keeps the segments). The default path does not reach any of it.
## Extra Notes
**How it was checked**
On v0.1.44 (this branch: #2398 + this commit; R9700 32 GB, ROCm 7.14.1, Unsloth UD-IQ4_XS, every expert in RAM, 262K
context, `--kv-grow --vram-elastic` through `serve.server`):
- Full HIP ctest: 111 of 112. The failure is `ple_parity`, which needs the Q2_0 fixture.
- Live:
- The cache was 10,940 slots (24.68 GiB), with `elastic K/V: 16384 of 262144 cells in VRAM`.
- `VRAM 4000` left 4,000 MiB free (9,427 slots holding experts).
- A 30.6K-token prompt with a needle was answered right while the cache was shrunk; the K/V grew from the cache's
slots.
- `VRAM 0` brought back 10,848 slots; the K/V keeps the rest for its cells.
- With setup's ROCm 7.10 wheels, #2398's check turns both switches off, the line says why, and the engine served
normally.
- CUDA: builds and links with nvcc 13.4.92, sm_120, 0 errors. Not run on an NVIDIA GPU.
- SYCL: `sycl/` builds with oneAPI DPC++ 2026.1, 0 errors (the port has its own `generate.cpp`; this change does not
reach it).
Measured before the rebase (0.1.43.1 + #2189, the same box), with a needle at 3/5 of each prompt and a 320-token answer:
| step | VRAM call | slots holding experts (of 10,865) | free | needle / decode |
|---|---|---|---|---|
| 78K prompt, whole cache | - | 10,492 (the K/V took 373) | - | ok / 96.6 tok/s |
| `reserve 4000` | 106 ms | 8,872 | 4,000 MiB | - |
| 78K prompt (the K/V grows again while shrunk) | - | - | - | ok / 92.0 |
| `reserve 8000` | 113 ms | 7,094 | 8,000 MiB | - |
| 156K prompt (the K/V to 2.13 GiB) | - | - | - | ok / 87.8 |
| `reserve 0` | 418 ms | 10,865 | 453 MiB | - |
| 78K prompt, then a short one (the K/V trims) | - | - | - | ok / 96.1 |
| `reserve 4000`, `reserve 0` | 101 ms, 199 ms | 9,281, 10,865 | 4,001, 453 MiB | - |
- The driver's own counter followed: 31.5 GiB, then 30.5 at `reserve 4000`, 28.0 after the 78K prompt, 24.1 at
`reserve 8000` with the 156K K/V, and back to 31.4.
- `--kv-grow` alone: every needle was found, and the K/V grew and trimmed as before.
- With the elastic image encoder (#2196, now in 0.1.44) on the same card:
- the first picture made `VRAM` keep 2 GiB free (52 ms), and the encoder started in 3.1 s;
- a 156K prompt grew the K/V to 163,840 cells while the encoder held its share (needle found);
- after 60 s without a picture the encoder stopped, and the cache was whole again.
**Not tested:** NVIDIA at run time, Windows, more than one shrink-grow cycle while the K/V holds a parked conversation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
En el sitio
Enlaces a install, modelos, releases.