Pull requests / #3288
#3288 [Engine] conversation snapshot: stage parked-state copies through pinned memory on Windows/WDDM
open · @denis-1337 · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
Issue: Resolves #3036 Supersedes #3037 (closed by the tracker reset; its head commit became unreachable when the branch was rebased on `main`, so GitHub could not reopen it - the reopen check itself passed). ## Problem and result On Windows/WDDM, parked-conversation snapshot copies at `--max-context 262144` are refused or corrupted: the engine logs `checkpoint save: conversation snapshot running-state copy: invalid argument`, and every later request returns non-finite (NaN) output. After this change the synchronous snapshot copies are staged through a real pinned buffer, and the same workload runs clean on the same rig. ## Related discussion #3036 (reproduced on 0.2.0-beta and 0.1.45). Bug fix, no defaults changed. Review feedback from @midhatn applied in the second commit. ## Approach All synchronous running-state copies in `conversation_state.cpp` are staged first through a pinned 16 MiB buffer (device→pinned via the copy engine, pinned→destination via memcpy), because on Windows/WDDM the HIP runtime misclassifies C++ heap destinations as pinned host memory: the direct copy is refused, or when accepted it interleaves with expert H2D streaming on the copy engine and corrupts the bytes (decode then reads garbage as expert weights and the grouped GEMM writes NaN). The save call also passes the copy direction explicitly instead of relying on pointer classification, and the sticky `invalid argument` is consumed and retried once. `STRATA_CKPT_STAGE_FIRST=0` restores the old path for A/B. Linux is unaffected (the direct path works there; staging is harmless). Cost: one memcpy hop per snapshot copy; measured decode unchanged (~55 tok/s). ## Validation Tested on `main` @ e88ee39d (0.2.0-beta, engine 0.1.45.3), Windows 11 Pro build 26300, AMD RX 9070 XT (gfx1201), driver 32.0.32015.2008, model IQ3_S, `--kv int8 --max-context 262144 --expert-cache auto --spec 4`, tuning table `tools/hip/gfx1201-hipblaslt-100500.txt`. The rebased branch file is byte-identical to the file built and tested here. **Passed:** - Official 0.2.0-beta install (prebuilt engine 0.1.45), 11 unique ~1.5k-token prompts via `serve/server.py` (park-unique script): `OK=0 ERR=11`, 43 non-finite events — bug reproduces on 0.2.0-beta. - HIP build of `main` @ e88ee39d, unpatched, same script: `OK=0 ERR=11`, nf=43 — bug present in the 0.2.0-beta source. - HIP build of `main` @ e88ee39d + this patch, same script: 2/2 rounds `OK=11 ERR=0`, 0 non-finite. - Production constellation (same config, port 1234): 22/22 requests OK, 0 non-finite since engine start, prompt-cache reuse active, decode ~55 tok/s. - On 0.1.45 (same rig, same scripts): baseline rounds nf=35/41/17 → patched 5/5 rounds nf=0; tuning-table constellation also clean. **Failed:** none observed with the patch applied. **Not run:** CUDA, SYCL and Metal builds — no CUDA/SYCL toolchain and no Apple hardware on this machine. The change uses only the CUDA-shaped API already used in this file (`cudaHostAlloc`, `cudaMemcpyAsync`, `cudaStreamSynchronize`, `cudaFreeHost`); it compiles and runs on hardware under HIP. CUDA compilation stays unverified. The `cudaHostAlloc` failure path and the stream-sync failure path are inspection-only (no fault injection), as noted by @midhatn in review. ## Limits Staging adds one memcpy hop per snapshot copy (a few ms per parked conversation). `STRATA_CKPT_STAGE_FIRST=0` restores the old path for A/B. The misclassification itself lives in the WDDM/HIP layer; this PR avoids it on the checkpoint path rather than fixing the classification. Note: this issue and the accompanying PR were prepared by an AI agent (Qwen3.8-Flash-Next IQ3_S, served by Strata itself) together with the account owner.
Sur le site
Liens install, modèles, releases.