Pull requests / #2569
#2569 batch: a slot that gives its whole conversation back to the main session hands over its checkpoints too
closed · @andyhrc · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux
Description
## Title Issue: none open that I found. ## Summary With `--batch`, a request's conversation moves between a batch slot and the solo path (the main session): a request left alone in its slot goes back to the solo path, and goes back into a slot when another request arrives. When a slot gives its **whole** conversation back to the main session, the main session's checkpoints are cleared and only the slot's tokens come over - the slot's own checkpoints (the turn boundary saved when the request was admitted) stay behind. When the conversation then goes back into a slot or is parked, there is no checkpoint left to copy. A next turn that continues the prompt but not the whole reply - a client that rewrites the end of the answer it sends back - then finds nothing to resume from and is read from token 0. This PR lets the checkpoints come with the conversation, as they already do for a read that gave way (`BYIELD`), and as an admission and `park=` already copy the longest one into the slot. Independent of #2527 / #2528. ## Seen in use 3 x RTX A4000 16 GB (layer split), Qwen3.8-Flash-Next IQ3_S, `--batch 6` in 3 groups, two people with Claude Code. A 121,183-token conversation went slot -> solo -> slot while the other user's requests came and went: ``` strata batch: slot 0 takes 121183 tokens (copied in 1501.8 ms) <- admission, with its turn checkpoint strata batch: slot 0 gave back 121479 tokens of this conversation (all it holds) strata batch: slot 0 takes 121699 tokens (copied in 1240.6 ms) <- back into a slot: no checkpoint to copy strata serve: prompt 125512 tokens = 0 reused + 125512 read in 40477 ms ``` The next turn shared 121,483 tokens with the slot but split from it 300 tokens into the previous answer, so it could only resume from the turn boundary - which was gone. 40 s of prompt reading instead of about 4,300 tokens. ## Measured Same machine; a 27.5K-token conversation that is admitted into a slot, goes to the solo path when the other request ends, goes back into a slot when a third arrives, and finishes; its next turn sends the first answer rewritten after its first 8% (before any mid-reply checkpoint). The same server code, only the engine differs: | Engine | Next turn: tokens reused | read | time | | --- | ---: | ---: | ---: | | before | 0 | 27,651 | 10.9 s | | this PR | 27,499 | 152 | 0.69 s | ## What changed `src/program/generate.cpp` and its SYCL copy `sycl/src/program/generate.cpp`, where an idle slot gives back all it holds (`slot_ck == nullptr`): its checkpoints are appended to the main session's chain (each a prefix of the slot's tokens, so of `live`), with a fresh LRU stamp. The partial-read case (`BYIELD`), which already did this, is unchanged and not done twice. ## How it was checked - The scenario above on CUDA (sm_86, Linux), before and after; built with the regular native build. - Not built: SYCL (no oneAPI here) and HIP. The SYCL change is the same 11 lines in the same block. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.