#6537 [Engine] Keep evicted parked conversations on disk, also with --layer-split and --batch

open · @locomas-lab · 0 comentários · No GitHub

Multi-GPUAMD / HIP

Descrição

Problem: a parked conversation is lost when the RAM cache evicts it or the engine stops. SAVE/RESTORE is refused with --layer-split and --batch. We run both (3 x V100, --batch 4) and stop the engine on model switches.

Proposal, opt-in: `--conversation-disk DIR`, `--conversation-disk-gb N`. An evicted parked image goes to DIR (session format v2 = v1 payload plus one payload per stage; images without stages stay v1, byte for byte). A request matching no parked conversation reads the longest match back and uses the existing restore. Parked conversations are also written at QUIT.

Measured (3 x V100, layer split, cache slots 1, 30.7K-token conversations, 706 MB file, SATA SSD): cold 27.3-28.6 s; from disk 4.1-4.4 s; after engine restart 2.8 s.

Not run yet: batch concurrency, decode speed, HIP/SYCL.

Is this scope acceptable before a PR?

No site

Links install, modelos, releases.