#6546 [Engine] Opt-in resize_for_overwrite and parallel host memcpy for kv_mode != 0 park/restore
open · @YusenZhang0601 · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Problem and result For streamed / draft-ring KV (`kv_mode != 0`), parts `0..3` (`k`, `v`, `k_scale`, `v_scale`) live in pinned host memory (`st.host.*`), accounting for ~90% of a 100K-token conversation snapshot (~1.3 GB). Currently in `engine/src/core/conversation_snapshot.cpp`: 1. `conversation_kv_save()` only calls `dst[i]->resize_for_overwrite(sizes[i])` when `staged == true` (VRAM staging). For host-resident parts `0..3`, `staged` is false, so it falls back to zero-filling `dst[i]->resize(sizes[i])` — paying ~680–700 ms of single-threaded zero-fill + page faults on a cold 100K park before immediately overwriting every added byte. 2. Both `conversation_kv_save()` and `conversation_kv_restore()` / `conversation_kv_restore_prefix()` copy parts `0..3` sequentially through `transfer()` (per-segment synchronous CUDA/WDDM `cudaMemcpyAsync`), achieving only ~1.4 GB/s on Windows WDDM instead of host memory bandwidth. This PR adds an opt-in switch `STRATA_PARK_HOST_MEMCPY=1` (default OFF, with optional `STRATA_PARK_COPY_THREADS`, default `8`, clamped `1..16`) for `kv_mode != 0` that: - Calls the existing `ConversationBuffer::resize_for_overwrite(sizes[i])` for host-resident parts `0..3` (every added byte is overwritten immediately afterward); - Copies parts `0..3` in 4 MiB chunks via a persistent multi-threaded `HostCopyPool` using `std::memcpy` and `conversation_detail::background_thread_placement()`, while keeping part `4` (`idx_pooled`, device memory) on the existing VRAM async/sync transfer path. On my RTX A6000 48 GB workstation (Windows native WDDM, PCIe Gen3 x16, dual EPYC 7542 256 GiB DDR4, Qwen3.8-Flash-Next IQ3_XXS, `kv_mode=1`, 100K conversation ~1.3 GB K/V), measured on my `0.1.56` fork (`YusenZhang0601/[email protected]`, commit `a1c2c904139d5c6a028750141d7a75f35253b76a`, ABAB with 0 thermal throttling and 0 CPU interference): | Metric (100K conversation, `kv_mode=1`) | Baseline (`OFF`) | `STRATA_PARK_HOST_MEMCPY=1` (+ NUMA0) | `+ STRATA_PARK_ASYNC=1 + CKPT_ASYNC=1` (+ exact VRAM stage) | Speedup | |---|---:|---:|---:|---:| | Cold park `resize` | 682–692 ms | **73.8–74.5 ms (−89%)** | **0.32–0.53 ms (−99.9%)** | **1,290–2,150×** | | Cold park total (96K tokens) | 1,033–1,121 ms | **399–402 ms (−63%)** | **182–186 ms (−83%)** | **5.7–6.0×** | | Warm reuse park total | 126.5 ms | 124.6 ms | **15.7 ms (−87.6%)** | **8.1×** | | 100K restore | 288–298 ms | **240–255 ms (−15%)** | **184–230 ms (−23 to −38%)** | **1.3–1.6×** | | End-to-end cold conversation switch wall | 2,596 ms | **1,820 ms (−29.9%)** | **1,655 ms (−36.3%)** | **−941 ms** | ## Related discussion none ## Approach All changes are confined to `engine/src/core/conversation_snapshot.cpp`: - Add `park_host_memcpy_on()` (`STRATA_PARK_HOST_MEMCPY=1`, default OFF) and `park_copy_threads()` (`STRATA_PARK_COPY_THREADS`, default `8`, range `1..16`). - Add a lightweight persistent `HostCopyPool` (intentionally leaked on process exit like other worker pools so Windows CRT teardown cannot deadlock on parked threads; 0% CPU when idle; workers call `conversation_detail::background_thread_placement()` and `core::progress_beat()`). - In `conversation_kv_save()` when `fast_host && i < 4`: call `dst[i]->resize_for_overwrite(sizes[i])`, record `STRATA_PARK_TRACE` `resize` time and `resize_faults`, collect `(dst, src, n)` spans across `[keep, sizes[i])`, and execute `parallel_host_memcpy(host_spans)` once after the loop (so part `4` still uses `STRATA_PARK_ASYNC` / `STRATA_PARK_VRAM_MIB` unchanged). - In `conversation_kv_restore()` and `conversation_kv_restore_prefix()` when `park_host_memcpy_on() && st.kv_mode != 0` and no VRAM slot holds the image: parallel-copy host parts `0..3` back into `st.host.*`, transfer device part `4` via `transfer()`, and call `refresh_residency()`. Compatibility: - Default OFF: when `STRATA_PARK_HOST_MEMCPY` is unset or `0`, execution is identical to upstream and no worker threads are created. - Standard C++17 (`<thread>`, `<mutex>`, `<condition_variable>`, `<atomic>`, `std::memcpy`) plus existing `conversation_detail::background_thread_placement()` — no OS-specific headers added. ## Validation **Passed:** - `origin/main` (`6e38f588a`, engine `0.1.64`, commit `8cf92d5af` on `contrib/park-host-memcpy`) native Windows x86_64 build (MSVC `14.37.32822`, CUDA `13.0.48`, `sm_86`, CMake + Ninja `-DSTRATA_BUILD_CONVERSATION_TESTS=ON`): `ninja strata conversation_cache_test` PASS (`4.9s`, `strata.exe` `64,893,952 B`, SHA-256 prefix `98916dc3…`), `strata --version` -> `strata 0.1.64`; `conversation_cache_test` PASS (`rc=0`, `0.5s`: `13534 checks passed`; note that `conversation_cache_test` does not link `conversation_snapshot.cpp`, which is compiled and linked into `strata.exe`). - Historical ABAB and bitwise continuation parity on my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`, RTX A6000 48 GB `sm_86`, Windows WDDM, 2× EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, `kv_mode=1`): - Cold park `1,121 -> 186 ms` (with `PARK_ASYNC+CKPT_ASYNC`), `402 ms` without async; warm park `126.5 -> 15.7 ms`; restore `298 -> 184 ms`; - 100K cold park → restore 128-token greedy continuation (`reasoning_content + content`, 596 non-empty chars) matched `OFF` **100% bitwise** (`SHA-256 = 8a8e4a31144ef62fe4c39ad737e0e0c03a489ff918d7b518524ea6245501c813`); - Full-stack functional acceptance suite 10/10 PASS. **Failed:** - none observed **Not run:** - End-to-end GPU replay re-benchmark on `0.1.64` (`origin/main` `6e38f588a`) — the latency and parity measurements above were collected on `0.1.56` (note: the `0.1.56` test arm also enabled opt-in NUMA0 allocation for `ConversationBuffer`, which I plan to propose separately if desired). - 24,574-line `kvq` comparison (this change only copies raw K/V bytes between host buffers). - Linux, HIP, SYCL, and Metal builds (changes are pure standard C++17 inside `engine/src/core/conversation_snapshot.cpp`; SYCL maintains its own `engine/sycl/src/core/conversation_snapshot.cpp`). ## Limits - On cold first-time parks, the remaining ~150 ms of host copy time is bounded by Windows first-touch 4 KiB demand-zero page faults on newly allocated `ConversationBuffer` pages (warm buffer reuse drops to ~1–15 ms). - On multi-socket NUMA systems without NUMA-local allocation for `ConversationBuffer`, cross-socket page placement can still reduce host memcpy throughput; NUMA-local allocation is intentionally kept separate from this PR to maintain one problem per PR.
Sur le site
Liens install, modèles, releases.