#6355 [Engine] Layer-split serves cannot reach either disk cache: the SSD spill tier and session files are single-serve only
open · @marksmeltzer · 0 comments · View on GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
## The problem A machine that serves with `--layer-split` cannot reach either of 0.2.16's disk cache levels — the automatic SSD spill tier or the session save/restore files — even though these are exactly the machines whose conversations are largest and whose RAM overflows most. The RAM park tier supports layer split (a parked conversation holds one image per stage); the disk levels are single-serve only. So on a multi-GPU box, when the RAM cache fills, parked conversations are evicted outright, with no slower-but-huge level behind it. This matters because the boxes that need a layer split are the ones holding the biggest snapshots. Our serving box is one. ## What we measured (this box) Windows 11, 2×RTX 5090 (32 GB each), 128 GiB RAM, Strata 0.2.16, engine 0.1.62-windows-x86_64-cuda-cuda13, Qwen3.8-Flash-Next UD-Q4_K_XL, `--max-context 262144 --kv q8_0 --expert-cache auto --layer-split 24`, conversation cache `--conversation-cache-mib 20480 --conversation-cache-slots 8 --conversation-cache-min-free-mib 4096`, DFlash2 drafter. The 48-layer model does not fit one card, so the split is mandatory, not elective. From the engine log since the 0.1.62 start (about 12 hours, moderate agent traffic): - Startup says the split plainly: `layer split across 2 GPUs: CUDA0, then CUDA1 (split 24)`, `CUDA0 runs layers 0-23`, `CUDA1 runs layers 24-47, expert cache 21.46 GiB (7355 slots)`. - The RAM tier works across the split and is our most-used feature: 3,511 `parked` lines and 3,375 `restored` lines. Snapshots are 1.7–3.6 GiB each (`snapshot_bytes=17018229608`-class parked totals against a 20 GiB budget). - It already overflows at this load: the `evictions` counter reached 119, plus 6 `skip parking` lines. Every one of those is a full re-read when the conversation returns — the same cost #5895 measures at 67–78 s per 100K-token chat on smaller hardware. - The SSD spill tier never engaged: zero tier lines in the 156,914 `strata serve:` lines since that start, and the default folder `E:\Strata-data\conversation-spill` was never created. @lyrakat's run in #5895 shows what we're missing on hardware that *can* reach the tier: 110,115 tokens back from the SSD tier in 3.0 s against a ~73 s full read, same engine version, Windows, single card. ## What the code and docs say (unconfirmed static analysis — from reading 0.2.16 source and docs, not from an engine we modified) - The spill tier block in `generate.cpp` (around 13286–13326) is gated on `conversations.enabled() && stages.empty() && o.conversation_spill_mib != 0`, and the comment above it says why: *"Only on pressure: a park that fits in RAM never touches the disk. A single-serve only (a layer split's stage images are not written)."* A layer-split serve has one stage per GPU, so the block is skipped before the tier can even open or log. - `docs/DETAILS.md`, Session files section: *"Not supported with `--layer-split`, `--peer-device`, `--batch` or `--prompt-cache 0`. The engine has been run on Linux and CUDA only for this feature."* - The RAM side already represents a split conversation correctly: DETAILS.md — *"With `--layer-split` a parked conversation holds one image per stage."* So the gap looks like a snapshot-format limit, not a hardware one: the on-disk session file format describes one session, and a split conversation is two half-images that have to be written and restored together, on the same split. ## What we would like 1. **Spill tier: write one image per stage.** When a park overflows the RAM budget on a layer-split serve, write each stage's image to the tier folder as a set, and restore the set together. The in-RAM park already keeps exactly this representation, so the data structure exists; it needs a disk sibling. Restoring into a serve with a *different* split can stay refused with a clear log line. 2. **Session save/restore: same treatment afterwards.** `--slot-save-path` would then work on layer-split serves too. (Windows coverage beyond the current Linux+CUDA note would help as well — we are a Windows box and cannot test the documented-tested path.) 3. Nothing about the external-cache-control direction in #4763 changes — an outside manager still wants these APIs; this issue is about the APIs existing on split hardware at all. ## Limits of this report - Everything measured above is from our own box and its own log; we did not run a single-serve configuration on it, so we cannot quote what the tier would save here — #5895's single-card numbers stand on their own for that. - The code and documentation reading is static analysis on 0.2.16 as shipped; we did not build or patch the engine. - We have not tested `--peer-device` or `--batch` serves; they are named only because the docs name them beside `--layer-split`. Related: #5895 (the overflow problem, with measured single-card tier data), #4763 (externally controlled cache), #6081 (the park/eviction counters quoted above, already shipped).
Related on this site
Quick links to related pages.