Pull requests / #2728
#2728 STRATA_SLOT_COPY_DIRECT=1 (opt-in, CUDA): main session <-> batch slot copies go state to state, without a host image
open · draft · @andyhrc · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationLinux
Description
## Summary Every --batch admission reads its prompt on the main session and copies it into its slot (copy_to_slot); a slot that holds the start of a prompt is copied back first (copy_from_slot); a read that gives way and park= use the same copy. Each one saves a host image of every layer's K/V and of the running state and restores it: the K/V leaves its authoritative pool (VRAM, or the pinned host pool with --kv-resident) for pageable memory allocated for the purpose and goes back, with a device sync per layer. On a 3 x RTX A4000 layer split (--batch 6, --kv-resident 32768, 200K context) the engine log of one profile counts 1,346 copies into a slot (0.75 s on average, 89K tokens) and 969 copies back (0.80 s, 98K tokens): 30 minutes beside 6.1 hours of prompt reading, and no batch window runs meanwhile. ## Change (opt-in, CUDA) With STRATA_SLOT_COPY_DIRECT=1 the same bytes go from one session's state straight to the other's: - conversation_kv_copy(from, to, ...): conversation_kv_save + conversation_kv_restore without the image - both states' layouts at `upto` must be equal (nothing is written otherwise), the target's resident pages are then invalidated (streamed K/V) or refilled (the draft ring) by the code restore used (now restore_residency()); - conversation_checkpoint_copy(from, to, ids, ...): the running state device to device, and the spare pooled row and PLE window a restore rebuilds; refused before any write when the two sessions do not own the same state. copy_to_slot / copy_from_slot (and their draft K/V) use them; a slot checkpoint restore (copy_from_slot's `at`) is unchanged. Unset, every path is the one before. CUDA (src/); SYCL keeps its own copies and is not touched. ## Results Measured on our production build (this change with #2527 and #2528; 3 x RTX A4000, Qwen3.8-Flash-Next IQ3_S, 0.1.44.1, --batch 6 in 3 groups), the same binary off / on, one filler request keeping batch mode on: | Tokens | into a slot (off / on) | back to the main session (off / on) | | ---: | ---: | ---: | | 430 | 134 / 70 ms | 141 / 77 ms | | 8.7K | 211 / 106 ms | 146 / 76 ms | | 46K | 702 / 250 ms | 518 / 122 ms | | 117K | 1,756 / 489 ms (next turn 1,399 / 152 ms) | 1,506 / 216 ms | | 37K (park=) | 412 / 255 ms | | A 117K conversation's next turn spent 1,751 ms before decoding, 444 ms with the variable. Single user 64.6 / 63.1 tok/s, 6 concurrent x 2048 176.7 / 178.1 tok/s (tools/serve_load.py). ## How it was checked Exactness: conversation_snapshot_test (4,670 checks, GPU): every K/V format and mode at 9 lengths equals save + restore byte for byte, resident pages as a restore leaves them, another layout refused with nothing written; a whole session's running state the same. With docs/BATCHING.md's exactness settings (STRATA_IQ_MT_MIN=1, --pcie-frac 0, --adapt-every 1000000, --no-prefill-borrow) the replies of both turns at 8.7K / 46K / 117K tokens in batch mode are the same tokens off and on; a park and restore gives the identical greedy reply either way. Built natively on CUDA (sm_86, Linux): the engine and `conversation_snapshot_test`. Not built here: HIP (it shares `src/` with CUDA; the new code uses only the calls these files already use) and SYCL (not touched: it keeps its own copies of these functions). ## Extra Notes Independent of #2527 / #2528: `copy_to_slot` / `copy_from_slot` are upstream code that every `--batch` admission runs. We run it in production with those two on the machine above since 2026-10-16. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.