#6533 [Engine] A branching follow-up re-reads the whole prompt when the final turn is longer than 1024 tokens

open · @rriggs · 0 comentários · No GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quants

Descrição

## Problem

A request that branches off a conversation the engine already holds re-reads the
whole prompt. The shape is the one a parallel agent makes: the same history sent
again with the final turn replaced — another question, or a different tool result
— so the two prompts share everything up to the start of that last turn.

The engine can only reuse a prefix from a checkpoint, so a branch needs a
checkpoint at the **message boundary**: the turn before the one the prompt ends
with. Two things stop it existing or reaching the slot:

1. `message_checkpoint_boundary()` scans back from the new assistant turn and
   gives up after `max_tail`, which defaults to **1024**. The scan crosses the
   final user turn, so any final turn longer than 1024 tokens hides the boundary
   and no checkpoint is made. Long final turns are ordinary in an agent loop:
   tool results, fetched pages and pasted documents go there.
2. The admission carries only the **longest** checkpoint that prefixes the
   conversation. The longest one (the turn checkpoint) diverges before the
   boundary, so a request branching there cannot use it and the slot holds
   nothing that request can resume from.

Both engines have both shapes. On the CUDA/HIP side
`engine/src/program/generate.cpp:20572` makes the same four-argument call
(`max_tail` 1024) and `:22744` keeps only the longest checkpoint on admission.
On the SYCL side the same two are at
`engine/sycl/src/program/generate.cpp:11317` and `:12856`. **The CUDA/HIP half is
reading, not measurement** — I have no NVIDIA hardware; a CUDA owner should
confirm it.

## Reproduction and measurement

Measured on an Intel Arc Pro B70 (SYCL engine, one card, 34 GB), IQ3_S,
engine 0.1.63, `--batch 2`, `--prompt-cache 6` (default),
`STRATA_CACHE_MESSAGE_BOUNDARY=1`, `--conversation-cache-mib 12288`.
Tree: `v0.2.17` (12d491da4). One prompt of **13,292 tokens** whose final user
turn is **1,658 tokens** (history before it is ~11.6k, so the boundary clears the
8192 `min_prefix`), sent twice, the second time with the final turn's text
replaced from its first token:

    request A (pristine)   prompt 13292 = 0 reused + 13292 read in 12842 ms, 1 checkpoints
    request B (pristine)   prompt 13292 = 0 reused + 13292 read in  7907 ms, 1 checkpoints
    request B (+ the two changes)
                           prompt 13292 = 11634 reused + 1658 read in 1606 ms, 2 checkpoints

So: before, the branch re-read every token; after, it re-read only the final turn
it changed. B's wall time 8.4 s → 2.1 s. Request A is unchanged in both runs
(0 reused, full read), and the extra checkpoint is only made when a boundary is
actually there and at least 8192 tokens in, so a cold or non-branching prompt
does no extra work.

The engine's own line is the measurement (`strata serve: prompt N tokens = R
reused + M read`), with its `1 checkpoints` / `2 checkpoints` counting the
checkpoints kept for the request.

## Proposed change

Two hunks, in `engine/sycl/src/program/generate.cpp` in the pull request that
goes with this issue:

* ask `message_checkpoint_boundary` for `max_tail` 16384 (a full turn) instead of
  the 1024 default;
* keep every checkpoint that prefixes the conversation on admission, not only the
  longest.

The cost is host RAM for the extra checkpoint copies, bounded by `--prompt-cache`:
for the conversation above the parked image went from 439,554,120 to 557,664,072
bytes (+118 MB, +27%) with the boundary checkpoint kept.

Relation to earlier work: #2518 (server-side slot choice) and #1906 (slot → solo →
slot) are the routing halves and are upstream. This is the engine half they route
to. #2566 was the earlier attempt at the whole set and was closed by the 0.2.0-beta
tracker reset, with review findings — this issue and its pull request are the
narrow version: one file, no `serve/server.py`, no #2384 port, and the carry-all
scoped to the admission.

## What was not run

* CUDA/HIP: not built, not run (no NVIDIA hardware here). Code reading only, as
  noted above.
* HIP and Metal: not built, not run.
* The `max_tail` half is what the measurement above exercises. The carry-all half
  needs a second chat occupying a batch slot while the branch arrives; it was not
  isolated here, so it is proposed on the code argument (the longest checkpoint
  diverges before the boundary) rather than on its own number.

No site

Links install, modelos, releases.