Pull requests / #1511

#1511 Trace prompt-cache costs and reuse snapshot buffers

open · @InB4DevOps · 0 评论 · 在 GitHub 查看

AMD / HIPNVIDIA / CUDA

描述

## Summary
- Add opt-in, log-only prompt-cache decision and phase tracing for CUDA/HIP and SYCL, including checkpoint allocation, GDN, PLE, and indexer timings and bytes.
- Add deterministic live workloads, trace analysis, and offline tooling tests.
- Reuse restored running-state host allocations for the next snapshot capture while refreshing every state byte.
- On branch switches that replace the active checkpoint chain, move that chain into the parked snapshot rather than deep-copying it. Keep the chain if admission or parking fails; preserve memory-budget accounting.
- Harden the trace runner against occupied ports, unloaded servers, paths with spaces, and backend-specific build directories.

## Performance
Measured on the local `branch_switching` workload using alternating settings on the same engine build:
- Running-state allocation reuse (first commit): total park capture 1025.0 ms -> 936.9 ms (-8.6%); eligible restored-branch parks 628.8 ms -> 540.8 ms (-14.0%).
- Checkpoint-chain move (second commit, with running-state reuse already enabled): five park captures totaled 939.0 ms -> 627.0 ms (-33.2%) across two off/on pairs. Reuse decisions and partial rereads were unchanged.
- Detailed trace of six checkpoint saves showed 174 ms spent allocating and 411 ms copying about 118 MB of GDN state per save; the chain-move optimization targets parking, not checkpoint creation.

## Verification
- `cmake --build build --target strata conversation_cache_test conversation_snapshot_test conversation_validation_test conversation_transfer_test`
- `./build/conversation_cache_test` (4,201 checks)
- `./build/conversation_snapshot_test` (3,985 checks)
- `./build/conversation_validation_test` (972 checks)
- `./build/conversation_transfer_test` (1,548 checks)
- `python3 -m unittest tools.test_analyze_prompt_cache_log tools.test_prompt_cache_workloads` (19 tests)
- `bash -n tools/prompt_cache_trace.sh` and `git diff --check`
- Live six-scenario workload (29 requests) completed with the checkpoint-chain move enabled.

SYCL source and tests are mirrored, but a SYCL toolchain was not available locally.

站内延伸阅读

链到安装、模型与版本说明,便于 SEO/GEO,非官方 issue 正文。