Pull requests / #2566

#2566 SYCL: Fix concurrent prefix cache

open · @rriggs · 0 comentarios · En GitHub

BenchmarksModels & quants

Descripción

# SYCL: Fix concurrent prefix cache

Issue: Resolves #2518, #1906 for SYCL

## Summary
Concurrent requests stop using the prompt cache: with --batch 2 or more, a client whose prompts share a prefix gets a full re-read on every request after the first, where --concurrency 1 reuses the cache. Three changes fix it: the message-boundary scan now reaches past a long final turn, each batch slot keeps the checkpoints its read takes, and an admission carries every checkpoint that prefixes the conversation instead of only the longest.

Builds on #2384 (the slot → solo → slot checkpoint fix) and ports it to SYCL; that and the two remaining failure shapes it left open are the ones here.


## What changed
`include/strata/program/message_boundary.hpp` (via the sycl/ copy's call site; the header is shared)
- `message_checkpoint_boundary()` scanned back from the new assistant turn-token for the previous turn-token, giving up after max_tail = 1024 tokens. A final user turn longer than that — a 2,048-token question over a shared context: a benchmark's depth cell, or a RAG request — meant the boundary between the shared context and the question was never found, so no checkpoint was taken there and every later request over the context re-read all of it. The call site now passes max_tail = 16384; the scan is token comparisons, so the cost is trivial.

`sycl/src/program/generate.cpp`
- Mirror (checkpoint_at): each checkpoint a batch slot's read takes is copied into that slot's own checkpoint list at the moment it is created. The main session's list is cleared by the next admission, and at --batch 4 a concurrent admission clears it before the first read finishes, so anything only in the main list can be lost mid-read. The mirror is set per BGEN admission and cleared when the read ends.
- Carry-all (BGEN admission): the admission kept one checkpoint in the slot — the longest prefix of the incoming prompt, which is the turn boundary (context + old question + reply). The message boundary between the shared context and the question is shorter and is what a later request branching inside the conversation resumes from; with only the turn checkpoint in the slot, that request diverged before the checkpoint's end and re-read the whole context. The admission now carries every checkpoint that prefixes the conversation.
- SYCL port of #2384: #2384's engine-side hunk was applied to src/program/generate.cpp only; the SYCL port builds its own copy (sycl/src/program/generate.cpp). Same hunk, applied there.


## Extra Notes
Measured on an Arc Pro B70 (32 GB), Flash-Next IQ3_S, --batch 4, --max-context 131072, four concurrent conversations of ~20.9K tokens each sharing context with distinct questions:
- all four follow-up requests resume at the message boundary (20,916–20,918 tokens reused, 22 read, ~460 ms each); the round takes 3.4 s against 72.5 s when they re-read
- llama-benchy --depth 8192 --pp 2048 --concurrency 4 with prefix caching: the depth cell went from 193 to 710 tok/s aggregate, and the variance dropped from ±57 to ±24
- at --concurrency 1 nothing changes: the reuse there came from the main session's live state, which was never broken     

The message-boundary checkpoint is behind STRATA_CACHE_MESSAGE_BOUNDARY=1 (opt-in, as before); the two new paths only matter when that is on. Nothing outside the listed files changes.

En el sitio

Enlaces a install, modelos, releases.