Pull requests / #2569

#2569 batch: a slot that gives its whole conversation back to the main session hands over its checkpoints too

closed · @andyhrc · 0 comments · View on GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

## Title
Issue: none open that I found.

## Summary

With `--batch`, a request's conversation moves between a batch slot and the solo path (the main session): a
request left alone in its slot goes back to the solo path, and goes back into a slot when another request arrives.
When a slot gives its **whole** conversation back to the main session, the main session's checkpoints are cleared
and only the slot's tokens come over - the slot's own checkpoints (the turn boundary saved when the request was
admitted) stay behind. When the conversation then goes back into a slot or is parked, there is no checkpoint left
to copy. A next turn that continues the prompt but not the whole reply - a client that rewrites the end of the
answer it sends back - then finds nothing to resume from and is read from token 0.

This PR lets the checkpoints come with the conversation, as they already do for a read that gave way (`BYIELD`),
and as an admission and `park=` already copy the longest one into the slot. Independent of #2527 / #2528.

## Seen in use

3 x RTX A4000 16 GB (layer split), Qwen3.8-Flash-Next IQ3_S, `--batch 6` in 3 groups, two people with Claude Code.
A 121,183-token conversation went slot -> solo -> slot while the other user's requests came and went:

```
strata batch: slot 0 takes 121183 tokens (copied in 1501.8 ms)      <- admission, with its turn checkpoint
strata batch: slot 0 gave back 121479 tokens of this conversation (all it holds)
strata batch: slot 0 takes 121699 tokens (copied in 1240.6 ms)      <- back into a slot: no checkpoint to copy
strata serve: prompt 125512 tokens = 0 reused + 125512 read in 40477 ms
```

The next turn shared 121,483 tokens with the slot but split from it 300 tokens into the previous answer, so it
could only resume from the turn boundary - which was gone. 40 s of prompt reading instead of about 4,300 tokens.

## Measured

Same machine; a 27.5K-token conversation that is admitted into a slot, goes to the solo path when the other request
ends, goes back into a slot when a third arrives, and finishes; its next turn sends the first answer rewritten
after its first 8% (before any mid-reply checkpoint). The same server code, only the engine differs:

| Engine | Next turn: tokens reused | read | time |
| --- | ---: | ---: | ---: |
| before | 0 | 27,651 | 10.9 s |
| this PR | 27,499 | 152 | 0.69 s |

## What changed

`src/program/generate.cpp` and its SYCL copy `sycl/src/program/generate.cpp`, where an idle slot gives back all it
holds (`slot_ck == nullptr`): its checkpoints are appended to the main session's chain (each a prefix of the slot's
tokens, so of `live`), with a fresh LRU stamp. The partial-read case (`BYIELD`), which already did this, is
unchanged and not done twice.

## How it was checked

- The scenario above on CUDA (sm_86, Linux), before and after; built with the regular native build.
- Not built: SYCL (no oneAPI here) and HIP. The SYCL change is the same 11 lines in the same block.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.