#6534 [Engine] sycl: keep the branch point a follow-up can reach

open · @rriggs · 0 Kommentare · Auf GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

## Problem and result

A request that branches off a conversation the engine already holds — the same
history with the final turn replaced, which is what a parallel agent sends — read
the whole prompt again. It reuses a prefix only from a checkpoint, and the one it
needs is the message boundary: the turn before the one the prompt ends with.

Two things kept that checkpoint from existing, or from reaching the slot:

* `message_checkpoint_boundary()` gives up after `max_tail`, default 1024. The
  scan crosses the final user turn, so a final turn longer than 1024 tokens (a
  tool result, a pasted document) hides the boundary and no checkpoint is made.
* The admission kept only the longest checkpoint that prefixes the conversation.
  The longest one diverges before the boundary, so it is useless to a request
  branching there, and the slot held nothing that request could resume from.

Measured on an Arc Pro B70, IQ3_S, engine 0.1.63, `v0.2.17` (12d491da4),
`--batch 2`, `--prompt-cache 6`, `STRATA_CACHE_MESSAGE_BOUNDARY=1`. One prompt of
13,292 tokens whose final turn is 1,658 tokens, sent twice with the final turn
replaced:

| | checkpoints | second request | read | prefill |
|---|---|---|---|---|
| before | 1 | 0 reused | 13,292 | 7,907 ms |
| after | 2 | **11,634 reused** | **1,658** | **1,606 ms** |

B's wall time 8.4 s → 2.1 s. The first request is unchanged in both runs (0
reused, full read).

## Related discussion

#2518 (server-side slot choice) and #1906 (slot → solo → slot) are the routing
halves and are in `main`. This is the engine half they route to.

#2566 was the earlier attempt at the whole set and was closed by the 0.2.0-beta
tracker reset; its review asked for (a) no unrelated hunks and (b) a host-RAM figure
for the carry-all. This PR is one file with nothing from `serve/server.py` or #2384,
and the figure above is the answer to (b).

#6533 The issue opened for this PR.

## Approach

`engine/sycl/src/program/generate.cpp`, two hunks, +12/−5:

* the `message_checkpoint_boundary` call asks for `max_tail` 16384 instead of the
  1024 default (a full turn), with `min_prefix` left at its 8192 default;
* the admission loop keeps every checkpoint that prefixes the conversation
  instead of only the longest. `!sl.img` moves from a post-loop guard into the
  condition; image conversations still carry none, as before.

The extra checkpoint is only made when a boundary exists and is at least 8192
tokens in, so a cold prompt or one with no branch behind it does no extra work.

Cost, measured: the parked image of the conversation above went from 439,554,120
to 557,664,072 bytes (+118 MB, +27%) with the boundary checkpoint kept. The number
of copies per slot is bounded by `--prompt-cache` (6 by default).

## Validation

Run on `v0.2.17` (12d491da4) built from source, and on the same tree plus this one
commit, bare metal on an Arc Pro B70 (one card, 34 GB), IQ3_S, engine 0.1.63,
`--batch 2 --prompt-cache 6`, `STRATA_CACHE_MESSAGE_BOUNDARY=1`.

**Passed:**

* the A/B above, both directions, same probe: the pristine build makes 1 checkpoint
  and the branch request reuses 0 of 13,292 tokens (7,907 ms of prefill); the
  patched build makes 2 checkpoints and the branch request reuses 11,634 of 13,292
  (1,658 ms), wall 8.4 s to 2.1 s;
* engine build: `ninja -C build -j6 strata` succeeds with `icx`/`icpx` oneAPI,
  `-DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_NATIVE_EXPERTS=ON
  -DSTRATA_GEOMETRY=flash_next`;
* the patch applies to `v0.2.17` with no other change, so the A/B isolates this
  commit: both arms are the same tree, rebuilt.
* `engine/src/core/conversation_cache_test.cpp` on the branch as it stands after
  main's advance (main `6e38f588a` plus these two commits):
  `conversation_cache_test: 13540 checks passed`, six of them new -- the whole
  chain is carried rather than only the longest, the order and lengths survive, a
  carried checkpoint is a copy of the state, a checkpoint as long as the prompt is
  not a prefix of it, a diverging checkpoint is dropped, and an image prompt
  carries none;
* RED on the old rule: the same six assertions run over main's selection (keep the
  longest prefixing checkpoint) pass 4 of 6, and the two that encode this change
  fail there.

**Failed:**

None observed.

**Not run:**

* **CUDA/HIP**: not built and not run (no NVIDIA hardware here). This commit does
  not touch `engine/src/program/generate.cpp`, but that engine carries the same two
  shapes, so it is not fixed by this PR -- see Limits.
* HIP and Metal: not built.
* The Python serve suite was not run against this tree, and would say nothing about
  the commit if it were: it touches one engine file and no Python. (A 2,114-test
  green run exists on a fork tree that carries this commit along with other local
  work, so it is not quoted here as validation of this change.)
* The carry-all hunk has no measurement of its own. Isolating it needs a second
  chat holding a batch slot while the branch arrives.
* The concurrent case the issue describes -- two conversations in one context and
  four branching follow-ups against `--batch 2` -- was not run.
* `--batch 2` was set for both runs, but the A/B itself sends its two requests one
  after the other.

## Limits

1. **The cap moved, it did not go.** A final turn longer than 16,384 tokens still
   hides the boundary and still makes no boundary checkpoint. 16,384 was chosen as
   a full turn, not as a bound that cannot be exceeded.
2. **Short conversations gain nothing, by design.** `min_prefix` stays at 8192, so
   a boundary less than 8192 tokens into the conversation is not checkpointed. The
   change costs nothing there and helps nothing there.
3. **Each conversation costs more memory.** Keeping the boundary checkpoint grew
   the parked image of the measured conversation by 118 MB (+27%, 439,554,120 to
   557,664,072 bytes). With `--prompt-cache 6` and a fixed conversation-cache
   budget, more conversations can now be evicted, which is exactly the case this
   change is meant to help. That pressure was not measured.
4. **The carry-all hunk is included on the code argument, not on a number.** No
   measurement isolates it, and the concurrent branch case was not run.
5. **This fixes SYCL only, and CUDA/HIP has the same two defects.** `engine/src/
   program/generate.cpp` still calls `message_checkpoint_boundary` with the
   four-argument default and still keeps only the longest prefixing checkpoint on
   admission, so a CUDA/HIP serve keeps reading the whole prompt on a branching
   follow-up. That is a second PR, and it wants hardware I do not have.
6. **One conversation, one measurement.** No run covers many conversations churning
   checkpoints, the conversation cache's disk tier enabled, or a soak; the two
   numbers above come from a single cold-then-branch pair.
7. **sl.checks.push_back(c) is a deep copy** of ids, imgs, gdn, ple, tails, dead, block_pos
   (conversation_cache.hpp:42). With the chain up to 6 deep, one admission can now copy
   six checkpoints where it copied one. The chain is bounded by --prompt-cache (6 by
   default: the root is pinned, the rest rotate by LRU), which bounds the copy count; the
   size of a single copy was not measured, so the slot-side peak is not quantified here.

Mehr auf der Site

Links zu Install, Modellen, Releases.