Pull requests / #3253

#3253 [Engine] session files work with --peer-device

open · @pspranger-throw · 0 comentarios · En GitHub

Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

## Problem and result

Supersedes #1273, which was closed in the 0.2.0-beta tracker reset. That PR cannot be reopened after its branch was rebased while closed (GitHub returns 422 on reopen); the diff and the validation below are unchanged.

The SAVE/RESTORE path refuses every session-file operation when `--peer-device` is set, so session persistence cannot be used on a two-GPU shape that keeps the second card as a peer expert cache. That shape is not an edge case: for the IQ3_S 262K configuration it is the only decode-positive lever.

The peer tier is a pure in-memory expert cache, rebuilt at load from the expert source files (which are already part of the model fingerprint). It holds no session state, so nothing is missing from a saved file and no per-run peer artifact can drift. After this change SAVE and RESTORE work normally with the peer active.

## Related discussion

#1273. Niko, 2026-10-17: "This lifts a refusal rather than speeding anything, so it is outside this speed pass; it applies cleanly and can be taken in the fixes pass."

The branch is rebased onto `main` (`e88ee39d`, head `99cf9d88`). Rebase-only: one line of code plus the DETAILS.md sentence; the diff is unchanged from the reviewed patch.

## Approach

- Remove the `--peer-device` refusal from the SAVE/RESTORE path. `--layer-split` stays refused: it moves K/V across stages, which a session file cannot represent.
- The peer tier needs no special save/restore handling: the session file holds the conversation state and the primary/streamed K/V pools, exactly as it does without a peer. The expert cache is rebuilt from the model files at load and only changes which experts are resident.
- The streaming restore's per-layer residency refill (`kv_stream_reset` / `kv_ring_restore`, from #1269) is unchanged and runs with the peer active.

## Validation

**Passed:**

- Full engine build on 0.2.0-beta source rebased onto `main` (`e88ee39d`), with #1269 applied for the streaming restore, CUDA 13.4, archs 75;86 (368 targets).
- `conversation_file_test`: 223/223 checks. All 8 conversation suites pass under `ctest`: conversation_cache, conversation_split_failure, conversation_memory, conversation_file, session_save_reclaim, conversation_snapshot, conversation_validation, conversation_transfer.
- Live drill on 0.2.0-beta (engine 0.1.45.3), IQ3_S 262K, `--peer-device 1`, 153K session, direct server on loopback:
  - peer tier active throughout: 3,339 experts / 6.34 GiB on the RTX 2060S, 189 MiB left free; primary held 8,662 experts (12,001 of 24,576 on the two GPUs);
  - SAVE: 153,426 tokens / 2.58 GB in 1.57 s;
  - RESTORE: 2.68 s; next turn 153,426 / 153,465 tokens reused (99.975 %);
  - fresh engine process + RESTORE: 2.79 s with MemAvailable 5,658 -> 5,308 MiB; next turn 153,538 / 153,574 tokens reused (99.977 %);
  - no `SERR`, no `FATAL`, no refusals.
- Prior live evidence on 0.1.45 (same change, older base), peer active:
  - shallow cycle: 10,462-token session saved through the slot API (396 MB, 314 ms), engine restarted, restored in 423 ms, next turn 10,495 / 10,515 tokens reused (99.8 %);
  - deep cycle: 198,830-token session (3.05 GiB) saved in 1.4 s, restored in 3.5 s, next turn 198,859 / 198,879 tokens reused (99.99 %) where the cold read is ~177 s;
  - gateway switch: 39,060 tokens / 833 MB saved in 959 ms, restored in 1.0 s, 99.95 % reused.

**Failed:**

- none observed.

**Not run:**

- HIP, SYCL and Metal builds, and Windows/macOS; only the CUDA/Linux build was run here.
- `--layer-split` remains deliberately unsupported and was not tested.

## Limits

- The peer's resident expert set is not saved. It is rebuilt at load from the expert source files; a different VRAM budget or a different peer card only changes which experts are resident, not the restored conversation or its K/V.
- The existing session identity and config-fingerprint gates still apply: a file saved with another engine configuration is refused, as before.
- This change only lifts the refusal for session files. `--layer-split` continues to return the existing refusal.

En el sitio

Enlaces a install, modelos, releases.