#3945 [Engine] serve: a park moves the checkpoint buffers it already holds instead of copying them, same image (opt-in, `STRATA_CKPT_PARK_REUSE=1`)

open · @ruibeikaa · 0 comments · View on GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

## Problem and result

**Two commits now.** This PR's saving was measured with `--prompt-cache-uniform`, which had not been posted, so it is the first commit here. Least recent use keeps the root and the newest checkpoints, so a request that branches far back (a client's compaction request, for one) reads back to the root; with the flag the checkpoints stay about conversation / N apart and such a branch reads about one spacing again. Replayed through three days of real request logs (the 1,250 requests of 20K tokens or more), 24 checkpoints and 4 recent: nine compaction requests read 1,173 -> 69 s. More checkpoints cost more per park, which is what the second commit, the switch below, removes. Each is opt-in on its own; `--prompt-cache-uniform` is refused with `--prompt-cache-spread`.

**Measured on PH402 SKU 200 boards: four GP100 dies, 32 GB HBM2 each (128 GB in all), every expert resident in VRAM, no CPU share, no PCIe share. The change itself is not GPU-specific:** it is host-side conversation-cache bookkeeping, and the copies it removes are host-memory copies on every build.

Every park copies the outgoing conversation's whole checkpoint chain into the parked image (`copy_checkpoints_beating`), about 118 MB per checkpoint for Swift 1.5 over four stages. On the next park of the same conversation, `drop_superseded` (#342) frees the copy parked a turn back, whose checkpoints are mostly the same bytes that are now copied again. So a park's cost grows with `--prompt-cache N`, and most of it is copying bytes the cache already holds and is about to free. On a client whose turns resume from a checkpoint rather than from the live state (here: the assistant message sent back differs from the generated tokens), that park sits on every turn's path before the prompt read: in a 30-turn replay with 6 checkpoints the parks took 23.7 s of 119.8 s of append reads; with 16 checkpoints 27.6 s of 123.6 s, and of the 3.9 s more, 87% was the checkpoint copy and 12% freeing it again (the checkpoint captures and evictions during the reads did not change).

With `STRATA_CKPT_PARK_REUSE=1` a checkpoint carries the identity of its captured bytes, and a park moves buffers instead of copying them wherever it holds the same capture already: the checkpoints of the superseded parked copies it drops anyway, and - when a parked conversation or a borrowed prefix replaces the outgoing chain right after - the outgoing chain's own. The parked image is the same as the copied one. In the same 30-turn replay, on my earlier stack, 16 checkpoints with the switch: append reads 117.15 s (two runs), against 119.93 s for the default 6 checkpoints without it (-2.3%) and 123.62 s for 16 without it (-5.2%); the parks moved 1,520 checkpoint parts (43.2 GB) and copied 136. Every reply was identical to the same settings without the switch. Unset (the default) nothing changes.

## Related discussion

No scope issue: opt-in, no format or default change. It builds on #342's `drop_superseded` and #752's checkpoint split (both unchanged when the switch is off).

## Approach

- `--prompt-cache-uniform` (`include/strata/program/conv_cache.hpp`, `generate.cpp`): `conv_cache::uniform_victim` evicts the checkpoint whose removal leaves the smallest gap in tokens, never the root, the newest or the `--prompt-cache-recent N` (default 4) before it, a pinned one or before the tail rule. The restore is the existing checkpoint path. `STRATA_CKPT_UNIFORM=N` is the flag with `--prompt-cache N`, `STRATA_CKPT_RECENT=K` is `--prompt-cache-recent K`, for launchers that pass only the environment.
- `include/strata/core/conversation_cache.hpp`: `ConversationCheckpoint::capture`, a process-unique number per `conversation_checkpoint_save` (`src/core/conversation_state.cpp`). Copies keep it (`conversation_copy.hpp`), the layer split keeps each part's, a checkpoint assembled from a split prompt read takes its first part's, and one read back from a conversation file has 0, which never matches. Equal nonzero numbers mean equal bytes. Run-time only, like `pinned`; not written to files.
- `drop_superseded(..., harvest)`: with a harvest vector the dropped entries' checkpoints (every stage image's) are moved there instead of freed with the entry; without it (the default) nothing changes.
- `ConversationParkSource` (`conversation_snapshot.hpp`) + `conversation_detail::park_checkpoints` (`conversation_copy.hpp`): for each checkpoint of the image, move the buffers of a harvested checkpoint with the same capture, or of the view's own checkpoint when the caller consumes the chain, else copy as before. The metadata (ids, images, `used`, `pinned`, `capture`; no prompt plan, no stage parts) is set exactly as the copy sets it, so the image is the copy's. `conversation_snapshot_save` takes the source as an optional last argument (null = the copy).
- `src/program/generate.cpp` (`park_current_body`): with the switch, keep only harvested checkpoints that a view still holds (the rest are freed at once, as without it), pass the sources to the stage images' saves, and let the RAM admission ask only for the bytes it allocates. The chain is consumed only where the caller replaces it right after: before a borrow and before taking a parked conversation (three call sites); the chain is cleared after the split's merge-back.
- `STRATA_CKPT_TIMING=1` (diagnostic, prints only): one stderr line per request with the park's phases (drop, split, capture, merge and free), moved / copied counts, and the checkpoints' capture and eviction times.
- Disk sessions (`SAVE` / `RESTORE`): a checkpoint read from a file has capture 0, so it is copied as today.

## Validation

**Environment note: measured only on PH402 SKU 200 boards - four GP100 dies of 48 SMs and 32 GB HBM2 each (128 GB in all), application clocks 715 / 1050 MHz - with every expert resident in VRAM (no CPU share, no PCIe share, no adaptive tier).** Nothing in the change depends on the card.

Rig: Windows 11, driver 581.80 (TCC), CUDA 12.9, VS 2022; build `-DSTRATA_EXPERIMENTAL_SM60=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DSTRATA_BUILD_CONVERSATION_TESTS=ON`, ggml at the pinned 3cf03257f. Model: Swift 1.5 IQ3_S GGUF.

**Passed:**
- This branch (main 870aedd9 + the two commits), CPU, `CUDA_VISIBLE_DEVICES=-1`: `conv_cache_test` PASS, `conversation_cache_test` 4,848 checks, including `drop_superseded` handing a dropped two-stage copy's four checkpoints over whole and the split and the merge keeping each part's capture; `conversation_file_test` 183; `conversation_split_failure_test` (8 / 12 / 16 cases at 1 / 2 / 3 stages); `conversation_memory_test` 33; `conversation_validation_test` 972 host-only checks.
- Build: sm_60 with `STRATA_EXPERIMENTAL_SM60` (`strata` and the conversation tests), clean.
- End to end, **this branch's own revision**, four GP100 dies, Swift 1.5 IQ3_S GGUF (dense matrices in Q8_0), MTP, `--layer-split 12,24,36`, `--pipeline-windows 2`, `--max-context 262144`, environment `STRATA_Q8_SM60=1 STRATA_MTP_Q8_SM60=1` and nothing else but the arm's switches, one fresh engine per arm, greedy:
  - Default path, none of the switches, against main 870aedd9 unchanged, six requests with `STRATA_STATE_HASH=1`: 6 of 6 replies and 6 of 6 sessions equal, every item.
  - A shorter replay than the one below: one conversation, a ~30K-token cold read, 12 append turns of ~1.2K tokens with 32 tokens out each, the assistant message sent back not being what was generated, so every turn resumes from a checkpoint and parks. **Every reply equal, 13 of 13, across every run** (default checkpoints, 16 uniform, 16 uniform with the switch). With the switch the parks moved 264 checkpoint parts (7.5 GB) and copied 48.

    | 16 uniform checkpoints | append reads | the 12 parks |
    |---|---:|---:|
    | switch off, pair 1 | 112.02 s | 5.55 s |
    | switch on, pair 1 | 110.95 s | 4.73 s |
    | switch off, pair 2 | 112.15 s | 5.67 s |
    | switch on, pair 2 | 110.88 s | 4.71 s |

    **Parks -15.9%, append reads -1.0%.** In pair 2, on an idle machine, every one of the 12 parks was faster with the switch. Three other runs of the same replay (default checkpoints and the switch) were left out of this table: the machine was not idle and they were about 3 s slower on the same work in both arms. The saving against the default 6 checkpoints is measured in the longer replay below, not here.
- End to end, four GP100 dies, my earlier stack, **not this branch's revision**: engine 0.1.47 (main 011c9e60) plus about 35 opt-in GP100 commits of mine, among them `--prompt-cache-uniform`; the 16-checkpoint arms ran with it (`STRATA_CKPT_UNIFORM=16`). Swift 1.5 IQ3_S, 1M context, `--layer-split 12,24,36`, MTP, conversation cache 32 GiB, one fresh engine per arm, the same replay of a coding-agent session in every arm: a 66K cold read, 30 append turns (1.2-2.8K tokens read, 32 out each), the client's compaction request (summary 1), the compacted conversation, one more turn of the original conversation, a second compaction request (summary 2) - 35 requests.

  | arm | checkpoints | switch | 30 append reads (read_ms summed) |
  |---|---|---|---:|
  | A (run 1 / run 2) | 6 (default) | off | 120,158 / 119,702 ms |
  | B (run 1 / run 2) | 16, uniform | **on** | **117,206 / 117,090 ms** |
  | C | 16, uniform | off | 123,621 ms |

  - B against A: -2.32% (the four pairings -2.09..-2.55%), about 93 ms a turn; B against C: -5.2%. A's two runs differ by 0.38%.
  - Bitwise: B = C on 35/35 replies (and B = B, A = A on 35/35 between runs), so the switch changes no reply. B = A on 33/35; the two that differ are the compaction requests, which resume from a different checkpoint because 16 are kept (summary 1 read 45.86 s -> 8.22 s, summary 2 25.93 s -> 1.68 s, the same with or without the switch): that is the checkpoint count's doing, not this switch's.
  - The image built from the superseded copy's moved buffers was restored by the request that returns to the original conversation, and its reply equals A's and C's. The consumed-chain path ran (12 moves when the compacted conversation was parked) but was not followed by a hashed restore.
  - Host memory: the switch adds none (the engine's private commit after the replay 12.78 GiB with it, 12.79 GiB without it at 16 checkpoints).
- A pre-deployment check on the same stack, **not this branch's revision**: the stack's engine with this commit against the same engine without it, one fresh engine per arm, the 13 requests of a plain (non-replay) run (five short prompts, 8K / 26K / 88K-token contexts with 300-400 tokens of decode, 1K / 2K / 6.9K / 30K reads, a 151K read with 800 tokens of decode), the IQ3_S pack at `--pipeline-windows 3`: with every switch off, 13/13 reply hashes equal; with `STRATA_CKPT_PARK_REUSE=1` and 16 uniform checkpoints, 13/13 equal; on a second pack (Swift 1.5 IQ3_XXS, `--pipeline-windows 2`, 4 requests) with both on, 4/4 equal. Decode within -0.14% / -0.32% of the engine without the commit, whose own two runs differed by 0.21%; prompt reads equal; VRAM per die identical; engine RSS after the 13 requests 11.60 -> 11.67 GiB (the extra checkpoints, not the switch).

**Failed:** none observed.

**Not run:**
- HIP, SYCL and Metal builds (no such machine here; the change is host code shared by all of them and adds no kernel entry point). Any card other than GP100, though nothing in the change depends on it.
- **Any run of this switch with main's own eviction** (`--prompt-cache 16` or `--prompt-cache-spread`) or with the default 6 checkpoints alone: the switch does not depend on which checkpoints are kept, but its saving with main's settings was not measured.
- This switch together with `STRATA_PARK_ASYNC` or `STRATA_PARK_VRAM_MIB`.
- `conversation_snapshot_test` on this revision.
- A hashed restore of an image built by consuming the outgoing chain (see above).
- Disk sessions (`SAVE` / `RESTORE`) with the switch: their checkpoints have capture 0 and are copied as today; not run.

## Limits

- On when asked only.
- The sa

Related on this site

Quick links to related pages.