改动 / #3168
#3168 [Engine] kv: size the resident ring from the run's depth (opt-in, STRATA_KV_RING_ADAPT)
open · @xyzzing · 0 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDA
说明
## Problem and result The resident K/V ring is one global value (`--kv-resident`), and because the KV pool is sized by it (the pool *is* the ring), a ring chosen for depth is paid by shallow runs. Measured on a 24 GB gfx1100 card at 1K with MTP: a flat 49,152 ring costs **−40.2 %** decode (draft acceptance 207/207 → 149-156/215), while without MTP the same change is **+3.0 %** — the main path is fine, the drafter is what loses (filed as #3166). This PR sizes the ring from the run's depth instead: `clamp(prompt + max_new rounded up to a page, 32,768, --kv-resident)`. Opt-in (`STRATA_KV_RING_ADAPT`), default behaviour unchanged. A shallow run keeps exactly today's ring; a deeper one grows into the cap. ## Related discussion - #2450 — the "keep the K/V in VRAM when it fits" setup rule this extends to the case where it does not quite fit. - #2630 — the streamed page pool's eviction policy; ring size is the first-order lever next to it on this card. - #3166 — the shallow-depth drafter/ring interaction this avoids by keeping the floor at the default ring. ## Approach - `qsa_kv_ring_adapt(g, cells_needed, cap)` in `src/core/layer.cpp` (+ the declaration in `layer.hpp`): rounds the needed cells up to a page, floors at `max(qsa_kv_resident_min(), 32768)`, caps at the configured `--kv-resident`. - The hook in `src/program/generate.cpp` runs **before** the session arena is sized and the graphs are captured, because `qsa_state_bytes` → `kv_pool_bytes` sizes the pool by slots. - The floor is the engine's historical default ring (32,768) on purpose: with the allocator minimum (20,480) the same patch measured **−7.3 % at 1K**. ## Validation Passed: - HIP build (gfx1100, ROCm 10.2 nightly, the pinned SDK): clean on the v0.1.45 base and on the `v0.2.0-beta` base with the same patch. - Paired one-shot cells on the v0.1.45 line (same binary A/B, 5 pairs @1K, 3 @32K, 3 @128K, 256 generated tokens, the deployment's env constant across arms): **1K −0.2 %, 32K +12.9 %, 128K +6.4 %**; prefill 2,348/2,330 and 2,406/2,408 tok/s; TTFT 14.3/14.3 s and 54.8/54.7 s. - Binary-neutrality control (the patched binary with the env unset, 1K): −1.1 %, inside the cell noise. Failed: none observed. Not run: - CUDA and SYCL builds (no such toolkit on this machine). The patch is backend-neutral core code (`layer.cpp`, `generate.cpp`), but I have not compiled it for those backends. - GPU re-validation on the `v0.2.0-beta` base: the patch re-applies and compiles there, but the numbers above are the v0.1.45 line — I am not claiming they transfer across the 777-commit release. - The `--serve` path: it keeps one main session plus parked snapshots, so there is no per-request session to size a ring for; per-request sizing there needs per-conversation KV pools (a follow-up, not this patch). ## Limits - Opt-in and `generate`-path only. - The 32,768 floor is a constant (the engine's default ring), not a configuration knob. - Because the pool is the ring, the ring cannot be resized within a session — only chosen when the session's arena is built.
本站相关内容
相关页面的快捷入口。