贡献 / #2384

#2384 Batch slots keep a conversation's turn checkpoints across slot -> solo -> slot moves (#1906)

open · @davidbhoward · 0 评论 · 去 GitHub 看

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

说明

Issue: Resolves #1906

## Summary
With `"parallel": 2` on a layer split (pipelined groups of one slot) and thinking on, most turns of a two-agent session read their whole history again. Two causes, both small:

1. A request that leaves its slot for the solo path and comes back (the hand-back to the solo path when a slot is left alone) loses the slot's turn checkpoints. The client's next turn comes back without the reply's thinking, so it only matches up to that turn boundary, finds no checkpoint, and reads from token 0.
2. `pick_slot` skips the held-prefix choice whenever groups are pipelined (`if gs:`), so the returning request can be admitted over the other conversation's slot. Groups of one slot have no pad rows and keep their conversation, so this now only applies to `gs > 1`, and a slot that matches up to its last turn boundary counts as holding the conversation.

## What changed
- `src/program/generate.cpp`: when a slot gives back all it holds, its checkpoints move into the main session's chain with the conversation (they stay valid, the main session holds the same tokens).
- `serve/server.py`: `if gs:` -> `if gs > 1:`, and `held_prefix` also counts a common start that covers at least half of what the slot holds.

Nothing else. Output is unchanged: this only decides which cached state a turn continues from.

## Extra Notes
This PR was written and posted by Claude (an AI assistant), at the request of the fork's owner, @davidbhoward.

Measured on 2x RTX 3090, layer split 25, IQ3_S, 262144, `"parallel": 2`, on `296b59d2` with and without this change. Two clients with `reasoning_effort: low`, 6 turns each on a ~31K history, history sent back without the thinking, two runs each:

| | turns that read the whole history (2 expected) | time reading |
|---|---|---|
| main | 10, 10 | 95 s, 95 s |
| this change | 2, 2 | 27 s, 28 s |

At 235-245K histories the same miss costs about 57 s per turn. Built and tested on CUDA only; the change is host code, but I did not build HIP or SYCL. The reproduction script is small and I can attach it if useful.

本站相关内容

相关页面的快捷入口。