Pull requests / #2896
#2896 STRATA_SLOT_COPY_DIRECT: the slot drafters K/V device to device too (--batch-mtp)
closed · draft · @andyhrc · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUNVIDIA / CUDAModels & quants
Beschreibung
With `STRATA_SLOT_COPY_DIRECT=1` (dd544364) an admission and a slot's resume copy the running state and the QSA K/V device to device. With `--batch-mtp` the slot drafters' K/V still went through a host image (`conversation_kv_save` + `conversation_kv_restore`) in `copy_to_slot` and `copy_from_slot`. This makes that copy go through `conversation_kv_copy` too when the switch is on; `conversation_kv_copy` already restores the draft ring's residency (`kv_mode == 2`) the way a restore does. Without the switch nothing changes. This is the one part of #2728 that upstream's `STRATA_SLOT_COPY_DIRECT` does not have; it replaces that PR, which I am closing. ### Measured 2x RTX A4000 (layer split on GPUs 1 and 2), Qwen3.8-Flash-Next IQ3_S, `--batch 4 --batch-mtp`, `STRATA_BATCH_MTP_SERIAL=1`, `STRATA_SLOT_COPY_DIRECT=1`. The same requests (a conversation of 8.7K, 36K and 112K tokens, three turns each, while another client decodes), upstream/main 477ac13c against this branch: | | before | after | |---|---|---| | 112K tokens: admission slot copy | 89.1 / 89.4 ms | 10.9 / 11.6 ms | | 112K tokens: whole admission (`slot N takes`) | 158.3 / 158.4 ms | 79.6 / 80.6 ms | | 112K tokens: give-back on the next turn | 96.4 / 96.4 ms | 17.6 / 17.5 ms | | 112K tokens: the next turn's prompt (19 tokens read) | 484 / 490 ms | 406 / 410 ms | | 36K tokens: admission slot copy | 13.9 / 13.8 ms | 5.1 / 6.1 ms | | 36K tokens: give-back | 36.6 / 20.9 ms | 21.0 / 12.3 ms | | 8.7K tokens: admission slot copy | 10.2 / 10.2 ms | 8.0 / 8.1 ms | | 8.7K tokens: give-back | 15.2 / 11.6 ms | 9.4 / 2.2 ms | The slots' MTP acceptance is the same over the same answers (66.2-71.4 % before, 61.5-67.9 % after; the drafts depend on the copied K/V, so a wrong copy would show here first). The rest of the admission (about 60 ms) is the turn checkpoint's host copy, which `STRATA_SLOT_CKPT_MOVE=1` / `STRATA_CKPT_POOL=1` address. For context, on this box `--batch-mtp` with one serial group stays behind the pipelined groups without slot drafts at every concurrency (2 GPUs, 1/2/4/6 clients: 43.0/52.3/54.6/53.3 against 46.4/60.1/75.8/73.0 tok/s; 3 GPUs: 62.1/83.4/89.9/88.6 against 62.7/94.8/136.4/170.9), so the change matters where `--batch-mtp` is the batch path (one GPU, no layer split) or chosen anyway. ### Tests Built natively (CUDA 86); the 2-GPU server runs above. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Mehr auf der Site
Links zu Install, Modellen, Releases.