#6545 [Engine] Size STRATA_PARK_ASYNC VRAM stage by device-resident bytes under kv_mode != 0

open · @YusenZhang0601 · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

## Problem and result

When `STRATA_PARK_ASYNC=1` (or `STRATA_PARK_VRAM_MIB`) is enabled together with streamed or draft-ring KV (`kv_mode != 0`), `conversation_snapshot_save()` in `engine/src/core/conversation_state.cpp` computes the required VRAM staging size `kv_need` using `conversation_kv_bytes(...)`, which sums all 5 K/V parts (`k`, `v`, `k_scale`, `v_scale`, `pooled`).

However, under `kv_mode != 0`, parts `0..3` live in pinned host memory (`st.host.*`) and are **never** staged into the VRAM staging buffer by `conversation_kv_save()` (which checks `st.kv_mode == 0 || on_device(src[i])`). Only part `4` (`idx_pooled`, and `0` bytes for the draft layer which has no indexer) is actually device-resident and staged in VRAM. As a result, `conversation_park_async_begin(kv_need)` over-reserves VRAM by roughly **9×** (~1.5–1.6 GiB instead of ~160 MiB for a 100K-token conversation), eating ~1.3 GiB of unnecessary VRAM or failing the headroom check and falling back to synchronous parking.

This focused fix computes `kv_need` from device-resident bytes only (`parts[4]` via the existing `conversation_kv_part_sizes()` when `kv_mode != 0`, and unchanged `conversation_kv_bytes()` when `kv_mode == 0`).

On my RTX A6000 48 GB workstation (Windows WDDM, PCIe Gen3 x16, 2× EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, 262K INT8 KV, `kv_mode=1`, 100K conversation), measured on my `0.1.56` fork (`YusenZhang0601/[email protected]`, commit `a1c2c904139d5c6a028750141d7a75f35253b76a`):

| Metric (100K conversation, `STRATA_PARK_ASYNC=1` + `STRATA_CKPT_ASYNC=1`, `kv_mode=1`) | Before (full `conversation_kv_bytes`) | After (device-only `parts[4]`) | Delta |
|---|---:|---:|---:|
| Additional VRAM reserved for park + checkpoint async staging | ~1,600 MiB | **306 MiB** | **−1.3 GiB reclaimed** |
| Steady-state minimum free VRAM (`mem_free_min`) | ~4,437 MiB | **5,731 MiB** | **+1.29 GiB headroom** |

## Related discussion

none

## Approach

In `engine/src/core/conversation_state.cpp`, replace the direct `conversation_kv_bytes(...)` summation for `kv_need` with a small local helper `stage_bytes(q, idx)`:

- If `q.kv_mode == 0`: return `conversation_kv_bytes(q, g, upto, idx)` (unchanged).
- If `q.kv_mode != 0`: call the existing backend-agnostic `conversation_kv_part_sizes(q, g, upto, idx, parts, ignore)` and return `parts[4]` (`idx_pooled`, which is `0` when `idx == false` for the draft layer). If `conversation_kv_part_sizes` fails, fall back conservatively to `conversation_kv_bytes(...)`.

Compatibility:
- Default behavior (`STRATA_PARK_ASYNC` unset and `STRATA_PARK_VRAM_MIB` unset) is 100% unchanged because `conversation_park_async_begin()` is a no-op when both are off.
- Uses only the existing `conversation_kv_part_sizes()` symbol present across CUDA, HIP, SYCL, and Metal backends, so no cross-backend ABI or header changes are needed.

## Validation

**Passed:**
- `origin/main` (`6e38f588a`, engine `0.1.64`, commit `004529ef6` on `contrib/park-exact-vram`) native Windows x86_64 build (MSVC `14.37.32822`, CUDA `13.0.48`, `sm_86`, CMake + Ninja `-DSTRATA_BUILD_CONVERSATION_TESTS=ON`): `ninja strata conversation_cache_test` PASS (`16.9s`, `strata.exe` `64,886,784 B`, SHA-256 prefix `60d84524…`), `strata --version` -> `strata 0.1.64`; `conversation_cache_test` PASS (`rc=0`, `0.5s`: `13534 checks passed`; note that `conversation_cache_test` tests `conversation_cache.hpp` directly and does not link `conversation_state.cpp`, which is compiled and linked into `strata.exe`).
- Historical ABAB measurement on my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`, RTX A6000 48 GB `sm_86`, Windows WDDM, 2× EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, `kv_mode=1`, 100K conversation, 0 thermal throttling, 0 CPU interference): enabling `STRATA_PARK_ASYNC=1 + STRATA_CKPT_ASYNC=1` consumed only `306 MiB` total VRAM (`mem_free_min = 5,731 MiB` vs baseline `6,037 MiB`), 100K cold park → restore 128-token continuation was bitwise identical (`SHA-256 = 8a8e4a31144ef62fe4c39ad737e0e0c03a489ff918d7b518524ea6245501c813`, 596 non-empty chars), and full-stack functional acceptance passed 10/10.

**Failed:**
- none observed

**Not run:**
- End-to-end GPU replay re-benchmark on `0.1.64` (`origin/main` `6e38f588a`) — VRAM measurement above is from the equivalent logic on `0.1.56`.
- 24,574-line `kvq` comparison (this change only adjusts VRAM staging buffer allocation size).
- Linux, HIP, SYCL, and Metal builds (verified by static inspection that `conversation_kv_part_sizes` is defined in both `engine/src/core/conversation_snapshot.cpp` and `engine/sycl/src/core/conversation_snapshot.cpp`).

## Limits

- Still over-estimates slightly by the kept unchanged prefix of `idx_pooled` (as noted in the existing comment), which is minor (~160 MiB upper bound at 100K tokens vs ~1.5 GiB before).

Mehr auf der Site

Links zu Install, Modellen, Releases.