#5986 [Engine] conversation cache: a layer split borrows a parked conversation's prefix (opt-in STRATA_SPLIT_BORROW=1, #5612; 2x P100: main agent's return 33,317 -> 2,672 tokens read)
open · @Keoian · 0 评论 · 去 GitHub 看
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants
说明
## Problem and result On a layer split, a request that matches only a checkpoint inside a parked conversation takes the whole parked conversation (#5612). The usual case is a sub-agent that shares the main agent's system prompt and tools. The request then rewrites every cell past the match and the later checkpoints are dropped, so the main agent's next turn reads much of its history again. One GPU already borrows instead (#1164). With `STRATA_SPLIT_BORROW=1` a layer split borrows too: the request gets the K/V prefix from each stage's image and copies of the checkpoints on its path, and the parked conversation stays parked with all its checkpoints. Without the switch nothing changes. Measured on 2x Xeon E5-2696 v3 and 2x Tesla P100 16 GB (sm_60, `--layer-split 24`, both cards power-capped at 150 W), Qwen3.8-Flash-Next (a Q4_K_XL / IQ4_NL pack), `--max-context 262144 --kv k8v4 --spec 4`, MTP on, default `--prompt-cache 6` and 4 parked conversations. One server, requests one after another: a main agent of 12 turns (system prompt + 8 tools = 9,769 tokens, then about 2,400 tokens per turn), an unrelated chat, a sub-agent with the same system prompt and tools (2 turns), the main agent's next turn, and the same once more with a second sub-agent. Fresh engine per run; every request greedy with `max_tokens` 4. | request | without the switch (2 runs) | `STRATA_SPLIT_BORROW=1` | |---|---|---| | main agent returns after sub-agent 1 (40,411 tokens) | 21,727 reused + 18,684 read, 49.6 / 53.9 s | 37,656 reused + 2,755 read, 13.8 s | | main agent returns after sub-agent 2 (43,086 tokens) | 9,769 reused + 33,317 read, 104.8 / 109.7 s | 40,414 reused + 2,672 read, 13.3 s | | all 22 requests | 100,555 tokens read, 385 / 399 s | 53,981 read, 252 s | Every other request reused and read the same token counts in all three runs. The log shows two borrows (`borrowed 9769 tokens from a parked conversation, which stays parked, in 90.7 ms` and `70.4 ms`). ## Related discussion #5612 (the report and the analysis this follows; no maintainer comment there yet). #1164 added the one-GPU borrow. ## Approach - `borrow` is no longer gated on `stages.empty()` when the switch is set. - Before the outgoing conversation is parked, the donor is checked for every stage: each stage image validates against its stage's session, the stage images' checkpoints line up with the first stage's, a checkpoint at the match exists, and the copy-stream rule of a whole restore (#2409) holds on every card. If any of that fails the request does not borrow, the donor stays parked, and the prompt is read from what the session holds. - The restore calls `conversation_snapshot_restore_prefix` once per stage: the first stage's image with a new `first_stage` argument (the function otherwise refuses an image that has stage images, as before), then each `stage_images[i]` on its device. The draft layer's K/V comes from the image that holds it, as in the whole restore. - `conversation_checkpoints_borrow` (new, `conversation_cache.hpp`) copies the donor's checkpoints up to the match with every stage's part, in the form `conversation_checkpoints_merge` builds. The donor is not changed. - One GPU: the same calls with the same arguments as before (`stages.empty()` makes the new expressions equal the old ones). ## Validation **Passed:** - `conversation_cache_test` (built with `g++ -std=c++20 -Iinclude src/core/conversation_cache_test.cpp`): 4849 checks passed, with new checks for `conversation_checkpoints_borrow` (the copy, the donor unchanged, a checkpoint off the path, no checkpoint at the match, a stage short of a part, a part of another checkpoint). - CUDA build, `-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60`, CUDA 12.8, gcc 14, on 13eedf55 + this PR's engine commit. - The session above, three fresh engines (without, with, without): the table above. Two codes planted in the main agent's first message and fifth tool result came back right in all three. - The same three runs with `--adapt-every 0`, comparing the logprobs of the 4-token replies (top 5): - Against the second run without the switch, the run with it is identical on every request (largest difference 0) except the main agent's two returns. - Those two returns differ by 0.0204 and 0.0052 at most, with the same tokens. Without the switch the engine reads 18,684 / 33,317 tokens again in the batched prompt path; with it the conversation is restored as it was first read. - The first run without the switch left the other two at the main agent's turn 9, before any sub-agent arrived, and differs from both by the same amounts on six requests (up to 0.59 on turn 11; 0.0093 and 0.0035 on two sub-agent requests). So two runs without the switch differ from each other by more than the switch changes anywhere. - With adaptation on (the table's runs), the two runs without the switch gave identical logprobs, and the run with it started two replies (a sub-agent's) with other tokens, largest difference 0.546. I did not separate that further; the borrow run reads 46,574 fewer tokens before those requests, and the adaptive expert cache follows what was read. **Failed:** - The first version of this patch failed on its first borrow: `restoring a borrowed conversation prefix: conversation snapshot: a layer split's image restored as a single session`, and the engine exited. `conversation_snapshot_restore_prefix` refuses an image with stage images; the `first_stage` argument is the fix. All numbers above are from the fixed code. **Not run:** - HIP, SYCL and Metal builds (no such hardware or toolchains here). `conversation_state.cpp` and `generate.cpp` are shared, so those builds are unverified. - More than two stages, `--batch`, `STRATA_PARK_ASYNC`, `STRATA_PARK_VRAM_MIB`, `STRATA_CKPT_POOL`, `pin=N`, pictures, a control vector, and a split whose first stage holds the draft head. - The unit tests under `STRATA_BUILD_CONVERSATION_TESTS` other than `conversation_cache_test`. - The case where the donor cannot be borrowed from (stage images that do not line up, no copy stream): read from the code only, not triggered. ## Limits - Opt-in. One seed, one scripted session, one model and one machine. - A borrow copies the checkpoints on the path (here one, at the system prompt) with their stage parts, as the one-GPU borrow copies its own. - The parked main conversation needs its place in the cache: with the default 4 parked conversations, more agents than that still evict each other. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01N976Q2e2TARptyrfV8H8mz
本站相关内容
相关页面的快捷入口。