#6534 [Engine] sycl: keep the branch point a follow-up can reach
open · @rriggs · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
## Problem and result A request that branches off a conversation the engine already holds — the same history with the final turn replaced, which is what a parallel agent sends — read the whole prompt again. It reuses a prefix only from a checkpoint, and the one it needs is the message boundary: the turn before the one the prompt ends with. Two things kept that checkpoint from existing, or from reaching the slot: * `message_checkpoint_boundary()` gives up after `max_tail`, default 1024. The scan crosses the final user turn, so a final turn longer than 1024 tokens (a tool result, a pasted document) hides the boundary and no checkpoint is made. * The admission kept only the longest checkpoint that prefixes the conversation. The longest one diverges before the boundary, so it is useless to a request branching there, and the slot held nothing that request could resume from. Measured on an Arc Pro B70, IQ3_S, engine 0.1.63, `v0.2.17` (12d491da4), `--batch 2`, `--prompt-cache 6`, `STRATA_CACHE_MESSAGE_BOUNDARY=1`. One prompt of 13,292 tokens whose final turn is 1,658 tokens, sent twice with the final turn replaced: | | checkpoints | second request | read | prefill | |---|---|---|---|---| | before | 1 | 0 reused | 13,292 | 7,907 ms | | after | 2 | **11,634 reused** | **1,658** | **1,606 ms** | B's wall time 8.4 s → 2.1 s. The first request is unchanged in both runs (0 reused, full read). ## Related discussion #2518 (server-side slot choice) and #1906 (slot → solo → slot) are the routing halves and are in `main`. This is the engine half they route to. #2566 was the earlier attempt at the whole set and was closed by the 0.2.0-beta tracker reset; its review asked for (a) no unrelated hunks and (b) a host-RAM figure for the carry-all. This PR is one file with nothing from `serve/server.py` or #2384, and the figure above is the answer to (b). #6533 The issue opened for this PR. ## Approach `engine/sycl/src/program/generate.cpp`, two hunks, +12/−5: * the `message_checkpoint_boundary` call asks for `max_tail` 16384 instead of the 1024 default (a full turn), with `min_prefix` left at its 8192 default; * the admission loop keeps every checkpoint that prefixes the conversation instead of only the longest. `!sl.img` moves from a post-loop guard into the condition; image conversations still carry none, as before. The extra checkpoint is only made when a boundary exists and is at least 8192 tokens in, so a cold prompt or one with no branch behind it does no extra work. Cost, measured: the parked image of the conversation above went from 439,554,120 to 557,664,072 bytes (+118 MB, +27%) with the boundary checkpoint kept. The number of copies per slot is bounded by `--prompt-cache` (6 by default). ## Validation Run on `v0.2.17` (12d491da4) built from source, and on the same tree plus this one commit, bare metal on an Arc Pro B70 (one card, 34 GB), IQ3_S, engine 0.1.63, `--batch 2 --prompt-cache 6`, `STRATA_CACHE_MESSAGE_BOUNDARY=1`. **Passed:** * the A/B above, both directions, same probe: the pristine build makes 1 checkpoint and the branch request reuses 0 of 13,292 tokens (7,907 ms of prefill); the patched build makes 2 checkpoints and the branch request reuses 11,634 of 13,292 (1,658 ms), wall 8.4 s to 2.1 s; * engine build: `ninja -C build -j6 strata` succeeds with `icx`/`icpx` oneAPI, `-DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_GEOMETRY=flash_next`; * the patch applies to `v0.2.17` with no other change, so the A/B isolates this commit: both arms are the same tree, rebuilt. * `engine/src/core/conversation_cache_test.cpp` on the branch as it stands after main's advance (main `6e38f588a` plus these two commits): `conversation_cache_test: 13540 checks passed`, six of them new -- the whole chain is carried rather than only the longest, the order and lengths survive, a carried checkpoint is a copy of the state, a checkpoint as long as the prompt is not a prefix of it, a diverging checkpoint is dropped, and an image prompt carries none; * RED on the old rule: the same six assertions run over main's selection (keep the longest prefixing checkpoint) pass 4 of 6, and the two that encode this change fail there. **Failed:** None observed. **Not run:** * **CUDA/HIP**: not built and not run (no NVIDIA hardware here). This commit does not touch `engine/src/program/generate.cpp`, but that engine carries the same two shapes, so it is not fixed by this PR -- see Limits. * HIP and Metal: not built. * The Python serve suite was not run against this tree, and would say nothing about the commit if it were: it touches one engine file and no Python. (A 2,114-test green run exists on a fork tree that carries this commit along with other local work, so it is not quoted here as validation of this change.) * The carry-all hunk has no measurement of its own. Isolating it needs a second chat holding a batch slot while the branch arrives. * The concurrent case the issue describes -- two conversations in one context and four branching follow-ups against `--batch 2` -- was not run. * `--batch 2` was set for both runs, but the A/B itself sends its two requests one after the other. ## Limits 1. **The cap moved, it did not go.** A final turn longer than 16,384 tokens still hides the boundary and still makes no boundary checkpoint. 16,384 was chosen as a full turn, not as a bound that cannot be exceeded. 2. **Short conversations gain nothing, by design.** `min_prefix` stays at 8192, so a boundary less than 8192 tokens into the conversation is not checkpointed. The change costs nothing there and helps nothing there. 3. **Each conversation costs more memory.** Keeping the boundary checkpoint grew the parked image of the measured conversation by 118 MB (+27%, 439,554,120 to 557,664,072 bytes). With `--prompt-cache 6` and a fixed conversation-cache budget, more conversations can now be evicted, which is exactly the case this change is meant to help. That pressure was not measured. 4. **The carry-all hunk is included on the code argument, not on a number.** No measurement isolates it, and the concurrent branch case was not run. 5. **This fixes SYCL only, and CUDA/HIP has the same two defects.** `engine/src/ program/generate.cpp` still calls `message_checkpoint_boundary` with the four-argument default and still keeps only the longest prefixing checkpoint on admission, so a CUDA/HIP serve keeps reading the whole prompt on a branching follow-up. That is a second PR, and it wants hardware I do not have. 6. **One conversation, one measurement.** No run covers many conversations churning checkpoints, the conversation cache's disk tier enabled, or a soak; the two numbers above come from a single cold-then-branch pair. 7. **sl.checks.push_back(c) is a deep copy** of ids, imgs, gdn, ple, tails, dead, block_pos (conversation_cache.hpp:42). With the chain up to 6 deep, one admission can now copy six checkpoints where it copied one. The chain is bounded by --prompt-cache (6 by default: the root is pinned, the rest rotate by LRU), which bounds the copy count; the size of a single copy was not measured, so the slot-side peak is not quantified here.
Related on this site
Quick links to related pages.