Pull requests / #2728

#2728 STRATA_SLOT_COPY_DIRECT=1 (opt-in, CUDA): main session <-> batch slot copies go state to state, without a host image

open · draft · @andyhrc · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationLinux

Beschreibung

## Summary

Every --batch admission reads its prompt on the main session and copies it into its slot (copy_to_slot); a slot
that holds the start of a prompt is copied back first (copy_from_slot); a read that gives way and park= use the same
copy. Each one saves a host image of every layer's K/V and of the running state and restores it: the K/V leaves its
authoritative pool (VRAM, or the pinned host pool with --kv-resident) for pageable memory allocated for the purpose
and goes back, with a device sync per layer. On a 3 x RTX A4000 layer split (--batch 6, --kv-resident 32768,
200K context) the engine log of one profile counts 1,346 copies into a slot (0.75 s on average, 89K tokens) and 969
copies back (0.80 s, 98K tokens): 30 minutes beside 6.1 hours of prompt reading, and no batch window runs meanwhile.

## Change (opt-in, CUDA)

With STRATA_SLOT_COPY_DIRECT=1 the same bytes go from one session's state straight to the other's:
- conversation_kv_copy(from, to, ...): conversation_kv_save + conversation_kv_restore without the image - both
  states' layouts at `upto` must be equal (nothing is written otherwise), the target's resident pages are then
  invalidated (streamed K/V) or refilled (the draft ring) by the code restore used (now restore_residency());
- conversation_checkpoint_copy(from, to, ids, ...): the running state device to device, and the spare pooled row
  and PLE window a restore rebuilds; refused before any write when the two sessions do not own the same state.
copy_to_slot / copy_from_slot (and their draft K/V) use them; a slot checkpoint restore (copy_from_slot's `at`) is
unchanged. Unset, every path is the one before. CUDA (src/); SYCL keeps its own copies and is not touched.

## Results

Measured on our production build (this change with #2527 and #2528; 3 x RTX A4000, Qwen3.8-Flash-Next IQ3_S,
0.1.44.1, --batch 6 in 3 groups), the same binary off / on, one filler request keeping batch mode on:

| Tokens | into a slot (off / on) | back to the main session (off / on) |
| ---: | ---: | ---: |
| 430 | 134 / 70 ms | 141 / 77 ms |
| 8.7K | 211 / 106 ms | 146 / 76 ms |
| 46K | 702 / 250 ms | 518 / 122 ms |
| 117K | 1,756 / 489 ms (next turn 1,399 / 152 ms) | 1,506 / 216 ms |
| 37K (park=) | 412 / 255 ms | |

A 117K conversation's next turn spent 1,751 ms before decoding, 444 ms with the variable. Single user 64.6 / 63.1
tok/s, 6 concurrent x 2048 176.7 / 178.1 tok/s (tools/serve_load.py).

## How it was checked

Exactness: conversation_snapshot_test (4,670 checks, GPU): every K/V format and mode at 9 lengths equals save +
restore byte for byte, resident pages as a restore leaves them, another layout refused with nothing written; a whole
session's running state the same. With docs/BATCHING.md's exactness settings (STRATA_IQ_MT_MIN=1, --pcie-frac 0,
--adapt-every 1000000, --no-prefill-borrow) the replies of both turns at 8.7K / 46K / 117K tokens in batch mode are
the same tokens off and on; a park and restore gives the identical greedy reply either way.

Built natively on CUDA (sm_86, Linux): the engine and `conversation_snapshot_test`. Not built here: HIP (it shares
`src/` with CUDA; the new code uses only the calls these files already use) and SYCL (not touched: it keeps its own
copies of these functions).

## Extra Notes

Independent of #2527 / #2528: `copy_to_slot` / `copy_from_slot` are upstream code that every `--batch` admission
runs. We run it in production with those two on the machine above since 2026-10-16.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.