Pull requests / #1511
#1511 Trace prompt-cache costs and reuse snapshot buffers
open · @InB4DevOps · 0 comentários · No GitHub
Descrição
## Summary - Add opt-in, log-only prompt-cache decision and phase tracing for CUDA/HIP and SYCL, including checkpoint allocation, GDN, PLE, and indexer timings and bytes. - Add deterministic live workloads, trace analysis, and offline tooling tests. - Reuse restored running-state host allocations for the next snapshot capture while refreshing every state byte. - On branch switches that replace the active checkpoint chain, move that chain into the parked snapshot rather than deep-copying it. Keep the chain if admission or parking fails; preserve memory-budget accounting. - Harden the trace runner against occupied ports, unloaded servers, paths with spaces, and backend-specific build directories. ## Performance Measured on the local `branch_switching` workload using alternating settings on the same engine build: - Running-state allocation reuse (first commit): total park capture 1025.0 ms -> 936.9 ms (-8.6%); eligible restored-branch parks 628.8 ms -> 540.8 ms (-14.0%). - Checkpoint-chain move (second commit, with running-state reuse already enabled): five park captures totaled 939.0 ms -> 627.0 ms (-33.2%) across two off/on pairs. Reuse decisions and partial rereads were unchanged. - Detailed trace of six checkpoint saves showed 174 ms spent allocating and 411 ms copying about 118 MB of GDN state per save; the chain-move optimization targets parking, not checkpoint creation. ## Verification - `cmake --build build --target strata conversation_cache_test conversation_snapshot_test conversation_validation_test conversation_transfer_test` - `./build/conversation_cache_test` (4,201 checks) - `./build/conversation_snapshot_test` (3,985 checks) - `./build/conversation_validation_test` (972 checks) - `./build/conversation_transfer_test` (1,548 checks) - `python3 -m unittest tools.test_analyze_prompt_cache_log tools.test_prompt_cache_workloads` (19 tests) - `bash -n tools/prompt_cache_trace.sh` and `git diff --check` - Live six-scenario workload (29 requests) completed with the checkpoint-chain move enabled. SYCL source and tests are mirrored, but a SYCL toolchain was not available locally.
No site
Links install, modelos, releases.