#6465 [Engine] bound host RAM during conversation swaps
open · draft · @G0-0000 · 0 评论 · 去 GitHub 看
AMD / HIPNVIDIA / CUDADocumentationWindowsLinux
说明
## Problem and result A compatible whole-session RAM-cache take can hold the active host K/V, the incoming parked K/V, and another full parked image for the outgoing conversation. This PR adds an opt-in bounded host-memory exchange so two alternating conversations do not need that third full host-KV image on the supported path. `--conversation-cache-swap-block-mib 0` is the default and keeps the original path. Values 1-256 MiB enable the exchange. The option is documented in `docs/DETAILS.md`. ## Related discussion - Scope proposal: #6450. No maintainer scope decision has been received yet. - Related retained-buffer and RAM reports: #189, #2264, #5895. - Different approaches: #3351 / #1011 (shared pool), #4226 / #2510 (disk streaming). This PR does not implement disk streaming, compression, or expert streaming. ## Approach On a compatible ordinary whole-session RAM take, keep active host K/V addresses fixed and exchange K/V and scale bytes with one bounded scratch block. Reuse the incoming parked allocation for the outgoing conversation. A valid retained K/V prefix for the current conversation uses the original path. Borrow/prefix-only, device-only, batch/split, explicit restore, incompatible layouts and other unsupported cases retain their existing behavior. Ordinary `ConversationCache::put()` is unchanged. Exchange publication completes both queue insertions and potentially throwing pool reclaim before deleting superseded entries. Failures before exchange leave the old path intact; rollback preserves the old cache entries, serials and byte accounting. ## Validation Revision: fd7881bf2f0b769bc7f8f70d89aa3d2410b7aeef, based on upstream main 3142b66b96e9c0e775e30f620b11b191a1427c2a. Linux x86_64, CUDA 12.9.86, `70-real`, engine 0.1.63. Build directory: `/scratch/builds/upstream-chunked-conversation-swap-final`. **Passed:** - CUDA configure and build of `strata conversation_cache_test conversation_transfer_test conversation_snapshot_test conversation_segment_pool_test conversation_buffer_pool_test`: 168/168 completed. Configure used Release, CUDA, conversation tests, native experts, portable kernels, Q6K experts and experimental SM60; build used node 0 and `--parallel 2`. - `CUDA_VISIBLE_DEVICES=-1 ./conversation_cache_test`: 14,386 checks passed, including queue-insertion/pool-reclaim rollback and current-prefix gate cases. - `CUDA_VISIBLE_DEVICES=-1 ./conversation_transfer_test`: 2,111 host-only checks passed. - `./conversation_segment_pool_test`: 3,778 checks passed. - `./conversation_buffer_pool_test`: 21 checks passed. - `strata --conversation-cache-swap-block-mib VALUE --help`: 0, 1, 17, 63, 64, 255 and 256 exited 0; -1, 257, `64x`, `1,2`, overflow and missing values exited 2 with the expected diagnostics. Standalone help reports default 0/off. - Two independent static reviews of the final local and upstream diffs found no blocking issue. Ordinary `put()` matches the upstream baseline. **Failed:** None observed in the final targeted runs. **Not run:** - `conversation_snapshot_test` was compiled but not run; GPU runtime tests were not run. - HIP, SYCL, Metal, Windows and macOS builds/tests were not run. - A real upstream-revision service comparison with no option, explicit 0 and a positive value was not run. Default-path preservation is supported by the source diff and CLI checks, not byte-for-byte runtime-output testing. - Process peak RSS and end-to-end swap latency were not measured. The allocation high-water assertion was compiled as part of the snapshot test but that binary was not run. - No full test suite was run. ## Limits Only the compatible ordinary automatic whole-session host-RAM path is covered. No explicit file-restore, disk-cache, GPU-memory or expert-streaming behavior is changed. This PR makes no measured engine RSS or performance claim. A separate downstream V100 deployment passed smoke and four warmed A/B exchanges, but that was a different revision and is not counted as validation of this PR head. This remains a draft for scope review and the unrun checks above.
本站相关内容
相关页面的快捷入口。