Pull requests / #3168

#3168 [Engine] kv: size the resident ring from the run's depth (opt-in, STRATA_KV_RING_ADAPT)

open · @xyzzing · 0 comentários · No GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDA

Descrição

## Problem and result

The resident K/V ring is one global value (`--kv-resident`), and because the KV pool is sized by it (the pool *is* the ring), a ring chosen for depth is paid by shallow runs. Measured on a 24 GB gfx1100 card at 1K with MTP: a flat 49,152 ring costs **−40.2 %** decode (draft acceptance 207/207 → 149-156/215), while without MTP the same change is **+3.0 %** — the main path is fine, the drafter is what loses (filed as #3166).

This PR sizes the ring from the run's depth instead: `clamp(prompt + max_new rounded up to a page, 32,768, --kv-resident)`. Opt-in (`STRATA_KV_RING_ADAPT`), default behaviour unchanged. A shallow run keeps exactly today's ring; a deeper one grows into the cap.

## Related discussion

- #2450 — the "keep the K/V in VRAM when it fits" setup rule this extends to the case where it does not quite fit.
- #2630 — the streamed page pool's eviction policy; ring size is the first-order lever next to it on this card.
- #3166 — the shallow-depth drafter/ring interaction this avoids by keeping the floor at the default ring.

## Approach

- `qsa_kv_ring_adapt(g, cells_needed, cap)` in `src/core/layer.cpp` (+ the declaration in `layer.hpp`): rounds the needed cells up to a page, floors at `max(qsa_kv_resident_min(), 32768)`, caps at the configured `--kv-resident`.
- The hook in `src/program/generate.cpp` runs **before** the session arena is sized and the graphs are captured, because `qsa_state_bytes` → `kv_pool_bytes` sizes the pool by slots.
- The floor is the engine's historical default ring (32,768) on purpose: with the allocator minimum (20,480) the same patch measured **−7.3 % at 1K**.

## Validation

Passed:
- HIP build (gfx1100, ROCm 10.2 nightly, the pinned SDK): clean on the v0.1.45 base and on the `v0.2.0-beta` base with the same patch.
- Paired one-shot cells on the v0.1.45 line (same binary A/B, 5 pairs @1K, 3 @32K, 3 @128K, 256 generated tokens, the deployment's env constant across arms): **1K −0.2 %, 32K +12.9 %, 128K +6.4 %**; prefill 2,348/2,330 and 2,406/2,408 tok/s; TTFT 14.3/14.3 s and 54.8/54.7 s.
- Binary-neutrality control (the patched binary with the env unset, 1K): −1.1 %, inside the cell noise.

Failed: none observed.

Not run:
- CUDA and SYCL builds (no such toolkit on this machine). The patch is backend-neutral core code (`layer.cpp`, `generate.cpp`), but I have not compiled it for those backends.
- GPU re-validation on the `v0.2.0-beta` base: the patch re-applies and compiles there, but the numbers above are the v0.1.45 line — I am not claiming they transfer across the 777-commit release.
- The `--serve` path: it keeps one main session plus parked snapshots, so there is no per-request session to size a ring for; per-request sizing there needs per-conversation KV pools (a follow-up, not this patch).

## Limits

- Opt-in and `generate`-path only.
- The 32,768 floor is a constant (the engine's default ring), not a configuration knob.
- Because the pool is the ring, the ring cannot be resized within a session — only chosen when the session's arena is built.

No site

Links install, modelos, releases.