Issues / #3539
#3539 [Engine] session files are refused under --layer-split (501), so a split restart re-reads the whole conversation
open · @xunying7860 · 0 comments · View on GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux
Description
## Summary Session files (shipped in 0.1.40, #661) are the durable way to keep a conversation across a process restart, but they are refused whenever the engine runs with `--layer-split` / `--peer-device` / `--batch`: the server answers **501**, and `docs/DETAILS.md` states it explicitly — > "Not supported with `--layer-split`, `--peer-device`, `--batch` (the config's `"parallel"`, > #465; the server answers `501`) or `--prompt-cache 0`." Meanwhile the RAM conversation cache **does** work under the split since 0.1.39 (#747): "a parked conversation holds one image per stage" (`docs/DETAILS.md`, Conversation cache). So the per-stage state mechanism already exists — session files just do not use it. ## Why it matters (measured on a real box) The layer split is not optional for anyone whose model does not fit one card, and those are exactly the users who pay the most for a restart, because the conversation has to be re-read: | path | cost | |---|---| | restore a 329,625-token conversation from a session file | **18.6 s** (`session restored … in 18644.9 ms`) | | re-read instead (317,033-token prompt) | **~1,360 tok/s ⇒ ~4 minutes** of saturated GPU before the first token | Parking does not cover this case: it is host-RAM state and dies with the process. ## Request Support session files under `--layer-split` the way parking does — one file per stage, or one container holding a per-stage image — so that a restart with a split does not re-read the whole conversation. If a narrower first step is preferred: per-stage `save`/`restore` for `--layer-split` (same API, same 501 today) would already help. ## Environment * engine **0.1.44.1**, Linux (Ubuntu 22.04), CUDA build * 2× RTX 3080 20 GB, PCIe, **no NVLink / no P2P** * Qwen3.8-Flash-Next IQ3_S, kv int8, 1,048,576 ctx * `--conversation-cache-mib 30720 --conversation-cache-slots 2` * the numbers above come from the same engine + model on one card; the layer-split config is what we are moving to
Related on this site
Quick links to related pages.