Issues / #3539

#3539 [Engine] session files are refused under --layer-split (501), so a split restart re-reads the whole conversation

open · @xunying7860 · 0 コメント · GitHub で見る

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux

本文

## Summary

Session files (shipped in 0.1.40, #661) are the durable way to keep a conversation across a
process restart, but they are refused whenever the engine runs with `--layer-split` /
`--peer-device` / `--batch`: the server answers **501**, and `docs/DETAILS.md` states it
explicitly —

> "Not supported with `--layer-split`, `--peer-device`, `--batch` (the config's `"parallel"`,
> #465; the server answers `501`) or `--prompt-cache 0`."

Meanwhile the RAM conversation cache **does** work under the split since 0.1.39 (#747):
"a parked conversation holds one image per stage" (`docs/DETAILS.md`, Conversation cache). So the
per-stage state mechanism already exists — session files just do not use it.

## Why it matters (measured on a real box)

The layer split is not optional for anyone whose model does not fit one card, and those are
exactly the users who pay the most for a restart, because the conversation has to be re-read:

| path | cost |
|---|---|
| restore a 329,625-token conversation from a session file | **18.6 s** (`session restored … in 18644.9 ms`) |
| re-read instead (317,033-token prompt) | **~1,360 tok/s ⇒ ~4 minutes** of saturated GPU before the first token |

Parking does not cover this case: it is host-RAM state and dies with the process.

## Request

Support session files under `--layer-split` the way parking does — one file per stage, or one
container holding a per-stage image — so that a restart with a split does not re-read the whole
conversation. If a narrower first step is preferred: per-stage `save`/`restore` for
`--layer-split` (same API, same 501 today) would already help.

## Environment

* engine **0.1.44.1**, Linux (Ubuntu 22.04), CUDA build
* 2× RTX 3080 20 GB, PCIe, **no NVLink / no P2P**
* Qwen3.8-Flash-Next IQ3_S, kv int8, 1,048,576 ctx
* `--conversation-cache-mib 30720 --conversation-cache-slots 2`
* the numbers above come from the same engine + model on one card; the layer-split config is
  what we are moving to

関連リンク

インストール・モデル・リリースへのサイト内リンク。