#6524 [Engine] SYCL: enable the conversation cache's SSD tier (0.2.17's ConversationSpill)
open · @rriggs · 0 评论 · 去 GitHub 看
AMD / HIPNVIDIA / CUDAModels & quants
说明
## Problem and result
Conversation spill support was added to `0.2.17` but that feature addition omitted
support for the SYCL backend.
`0.2.17` added a conversation cache the engine can park whole conversations in, and
an SSD tier behind it (`engine/include/strata/core/conversation_spill.hpp`), so a
chat that outlives the RAM budget is written to disk and read back instead of being
re-read token by token. The header is engine-neutral, but only the CUDA/HIP engine
integrates it: **the SYCL engine has no `--conversation-spill-dir` or
`--conversation-spill-mib`, no spill on eviction and no restore. On an Arc, a chat
that leaves the RAM budget is simply lost.**
With the PR, SYCL gains full SSD conversation spill support.
On an Arc Pro B70 (one card, 34 GB), IQ3_S, `main` + these two commits, bare metal,
`--conversation-cache-mib 1024 --conversation-spill-dir <fresh dir>
--conversation-spill-mib 32768`, twelve distinct conversations sent, then the first
one again:
conversation cache: SSD tier in <dir>/<instance>, up to 32768 MiB
conversation cache: spilled 5547 tokens to the SSD tier (evicted from RAM) in 103.9 ms; on SSD 1, 305 MiB
... (8 spills, RAM budget 1 GiB)
conversation cache: read 5547 tokens back from the SSD tier in 207.6 ms
(this prompt starts with 5516 of them); on SSD 7, 1983 MiB
the returning conversation: prompt 5521 tokens = 5516 reused + 5 read in 639 ms
its first send: prompt 5522 tokens = 0 reused + 5522 read in 3559 ms
So the chat the RAM budget had to evict came back off the disk instead of being
re-read: 5,516 of 5,521 tokens reused, 3.6 s → 0.64 s of prefill. Nine session
files, 2.5 GiB, on disk afterwards.
## Related discussion
#5835
This is the SYCL half of the tier `0.2.17` added. The CUDA/HIP integration
(`engine/src/program/generate.cpp`, `conversation_disk.cpp`) is untouched and is the
reference this follows; where the two differ the commit messages say so.
## Approach
Same as for CUDA/HIP integration.
This implementation follows CUDA in being on unless `--conversation-spill-mib 0`, so a
SYCL server that wrote nothing to disk before will now spill.
Two deliberate deviations from the CUDA integration, both because the SYCL engine
has no equivalent:
* no `BackgroundWriter`: the evicting request pays the write synchronously
(~100 ms for a 5.5k-token conversation above) instead of handing it to a thread;
* no `pools_budget` shrink: this engine has no such pool.
The second commit is the read-limits fix the restore needs: the limits are formed
with the draft layer's K/V state when MTP is in use, exactly as the save path does
(`use_mtp ? &mtp.kv_state() : nullptr`); without it the limits cannot be formed and
the read-back is dropped.
`engine/sycl/src/program/generate.cpp` only, +190/−1:
* the header is included and its options parsed: `--conversation-spill-dir`
(default `<data>/conversation-spill` when `--pack` is under `<data>/packs`) and
`--conversation-spill-mib` (`-1` = the disk's free space decides, `0` = no tier),
with both in `--help`;
* the tier is opened at start-up with `spill_default_budget()`, folders left by
ended engines are cleaned, and a disk that is too full turns the tier off with a
line saying why, rather than failing the start;
* `conversations.set_evict_sink(...)`: a parked conversation the RAM budget evicts
is written to the tier instead of being dropped, logged with its token count and
the tier's totals;
* a conversation that exceeds the RAM budget outright is written straight to the
tier at the RAM floor rather than skipped;
* on a request, `spill` is consulted after the RAM cache (`best()` and
`take_serial()`), under the same `SessionReadLimits`, and a read-back replaces the
`incoming` image so the normal restore path takes it.
**Compatibility:** the two flags are new, so no existing command line changes
meaning. The tier's own default matches CUDA's, so the SYCL default path does change
(it will spill where it previously wrote nothing); `--conversation-spill-mib 0`
restores the previous behaviour exactly.
## Validation
Run on `main` (3142b66b9) + these three commits, bare metal on an Arc Pro B70 (one
card, 32 GB), IQ3_S:
**Passed:**
* engine build: `ninja -C build -j6 strata` succeeds (`icx`/`icpx` oneAPI,
`-DSTRATA_SYCL_AOT=bmg-g31 -DSTRATA_NATIVE_EXPERTS=ON
-DSTRATA_GEOMETRY=flash_next`);
* the disk-tier run above, both directions: 8 spills, one restore of 5,547 tokens,
5,516 of 5,521 reused on the returning chat, 9 session files on disk;
* the same check on the container image built from our fork (engine 0.1.63): 8
spills, restore in 187.2 ms, 5,516 of 5,522 reused;
* on that image, as a live server, a 5,547-token restore in 201.1 ms with 5,516 of
5,522 reused, and the returning conversation completed in 1.5 s;
* `engine/src/core/conversation_spill_test.cpp`, the header's own test, now built and
registered by this engine (`ctest -R conversation_spill_test`): 74 checks, 0 failed.
**Failed:**
None observed.
**Not run:**
* the rest of the engine tests were not run against this branch; the spill test above
is the only one, and the other engine-level checks are the build and the live runs.
* CUDA/HIP and Metal: not built (this commit touches the SYCL engine only).
* MTP is on in both runs (`--batch-mtp`, `--mtp <dir>`), so the draft-layer read
limits are exercised; a run with MTP off was not made.
* the restore was checked with `--conversation-cache-mib 1024`, which forces
eviction quickly. A production-shaped 8192 MiB budget was started and serves, but
the spill path was not driven under it.
* a layer split (`--gpu 0,1`), and two conversations churning the tier at once under
`--parallel 2`.
* cross-engine reuse: whether a file written by this engine can be read back by the
CUDA/HIP one, which shares the format but was not available to test.
## Limits
1. **The write is synchronous.** There is no `BackgroundWriter` in this port, so the
evicting or capturing request pays the write in its own latency: 96–160 ms measured
for 1.6–5.5k tokens, more for a longer conversation and more on a slower disk. The
CUDA/HIP engine hands the same eviction to a background thread. The read-back is
synchronous in both engines, so that half is not a difference.
2. ~**The shared test is not wired into this build.**
`engine/src/core/conversation_spill_test.cpp` (273 lines) covers the header and is
built for CUDA/HIP, but it appears nowhere in `engine/sycl/CMakeLists.txt` or
`engine/cmake/`, so the SYCL build neither compiles nor runs it — nor was it run
against this branch. Adding it to the SYCL test target is the follow-up; until then
the evidence is the build plus the live runs above.~
本站相关内容
相关页面的快捷入口。