#3350 [Engine] checkpoint save: "conversation snapshot running-state copy: out of memory" exits the engine (21 times in 4 h), while parking admission refuses the same conversation (0.1.45, 2x RTX 3080 20 GB, 62.7 GiB RAM, ~310K contexts)

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

While serving two alternating long conversations (~310K tokens each) on a two-gpu layer split, the engine exited **21 times in 4 h 08 min** (exit code 1, `2026-10-19 19:20:21` .. `23:02:57`), each time right after a conversation checkpoint save ran out of host RAM. Every exit was followed by the runtime reloading the model (~45 s) and the next request succeeding, so the service looked healthy throughout.

The engine log shows the same three lines each time:

```text
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed
```

and the runtime reports:

```text
[strata] the engine stopped unexpectedly (exit code 1). The engine exited (code 1) after it reported: saving a checkpoint part failed - please report it at github.com/Niko1221/Strata/issues with the log. The next request starts the engine again.
[strata] done: 0 tokens in 2 s (0.0 tok/s) (error, cancel=False)
[strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...
```

The part I did not expect: **the parking admission gate refuses the very same conversation moments earlier, on the same host, and it is right.**

```text
strata serve: conversation cache: skip parking (physical RAM admission; need 2915 MiB plus 1536 MiB floor, 2437 MiB available, source meminfo; evicted 0, 0 still parked)
strata serve: conversation cache: parked 310875 tokens in 190.5 ms; parked=1 bytes=6022875424 evictions=0 snapshot_bytes=6022875424 reused_kv_bytes=4740933120
```

So parking declines cleanly when the RAM is not there, and then the checkpoint writer copies anyway and takes the engine down with it. Numbers on this box at the time of the failure: engine RSS 51.7-54.8 GiB of 62.7 GiB RAM, one parked snapshot of a 310K-token conversation 5.61 GiB (6022875424 bytes). With `--conversation-cache-slots 3` that is a planned ~17 GiB of snapshots on the few GiB that are left.

**What I expected instead:** a conversation checkpoint is an optimization, the same way parking is. If the copy cannot be made, drop that checkpoint (or that conversation's entry) and serve the request, rather than exiting the engine. As it stands a refused optimization becomes a 45 s outage, every parked conversation is lost, and in-flight requests fail.

Client-visible damage in the same window: `done: 0 tokens in 2 s (0.0 tok/s) (error, cancel=False)`, and one request died after 1519 generated tokens.

Also worth knowing for anyone monitoring this: **systemd never sees it.** The reported unit stayed `active`, `NRestarts=0`, `Result=success` for the whole 4 h 08 min run, because the engine is a child of `serve/server.py` and the restart happens inside the runtime. The only reliable signal is `journalctl -b -u strata | grep -c "engine stopped unexpectedly"`.

### Smallest steps or command that reproduce it

1. Two-gpu layer split with long contexts, conversation caching on with more slots than the RAM can hold:

```text
--layer-split 25 --max-context 524288 --kv int8 --spec 4 \
--conversation-cache-mib 12288 --conversation-cache-slots 3 \
--conversation-cache-min-free-mib 1536 --session-save-reclaim
```

2. Serve two conversations of ~300-310K tokens alternately from two clients (mine came from two connections on the same LAN host). In my log the last three failures arrived exactly 6:00 apart.

3. The failures stop when the slot count fits the RAM. With `--conversation-cache-slots 1` and nothing else changed, the same two clients, the same ~310K requests: **13 requests over 13.5 min, 0 engine exits, 0 out-of-memory lines** (largest prompt 311,068 tokens). Service memory peak 58 GiB with 1.9 GiB swap, against 59.2 GiB + 6.8 GiB swap before.

There is one other exit in the same boot with the same wording, from a different path: `verify: instantiate: out of memory` also exits the engine (1 occurrence). Same shape, same expectation.

### Reproduction status

Reproduced locally

### Strata version

Source install, `v0.1.45`, commit `477ac13c`, `engine/BUILD.json` `"version": "0.1.45"` (`GET /props` reports `Strata 0.1.45`).

### Graphics card, memory and system

2 x RTX 3080 20 GB (sm_86), 62.7 GiB RAM (`MemTotal: 65728088 kB`), CachyOS, kernel `7.2.8-1-cachyos`, NVIDIA driver `615.71.09`, CUDA 13. `zram` for swap (62.7 GB, 1.2 G in use), since that matters below.

### Model, size and settings

Qwen3.8-Flash-Next `IQ3_S` (native pack), `--max-context 524288` (`--rope-scaling yarn --rope-scale 2.0`, trained 262144), `--kv int8`, `--spec 4`, `--mtp`, `--vision`, `--expert-cache auto`, `--layer-split 25`, `--vram-reserve-mib 512`, `--pcie-frac 0.20`, `--conversation-cache-mib 12288`, `--conversation-cache-slots 3`, `--conversation-cache-min-free-mib 1536`, `--session-save-reclaim`.

### Logs

The failure, with the admission refusal that preceded it (engine log, one occurrence):

```text
strata serve: conversation cache: skip parking (physical RAM admission; need 2715 MiB plus 1536 MiB floor, 3442 MiB available, source meminfo; evicted 0, 0 still parked)
strata serve: conversation cache: restored 306479 tokens (live) in 836.2 ms; parked=0 bytes=5029207144
strata serve: prompt 306705 tokens = 306479 reused + 226 read in 2119 ms (106.7 tok/s), 348 generated in 4309 ms (80.8 tok/s), drafts accepted 184 of 236, 6 checkpoints
strata serve: conversation cache: parked 307054 tokens in 220.0 ms; parked=1 bytes=5872999448 evictions=0 snapshot_bytes=5872999448 reused_kv_bytes=4678049664
strata serve: checkpoint save: conversation snapshot running-state copy: out of memory
strata serve: saving a checkpoint part failed
```

Exit codes for the whole boot: 21 x `(exit code 1)` on this path, 1 x `(exit code -15)` which was my own `systemctl restart`.

Snapshot sizes observed for 309-310K-token conversations: 5.61-6.02 GB (`bytes=6022875424`, `bytes=5872999448`).

### Checked against 0.2.0-beta / engine 0.1.45.3 (static analysis, not run)

I did not run 0.2.0-beta on this box, so this part is from the source, not from a measurement:

- `src/core/conversation_state.cpp` is byte-identical between the `v0.1.45` tree and `v0.2.0-beta` (`diff` is empty), so the copy that fails is the same code.
- The call site in `src/program/generate.cpp` is unchanged in behaviour: `if (!checkpoint_save(part, stage_ss, g)) { e = "saving a checkpoint part failed"; return false; }`, still propagating to the caller's `if (!saved) { e = "saving a conversation checkpoint failed" + ckpt_why; return false; }`.
- Nothing in the 0.2.0 release notes mentions conversation-cache checkpoint memory.

If a memory admission check for this path did land somewhere I did not look, I would rather be corrected than file a duplicate, and I will happily re-test on a 0.2.0-beta build.

### One thing that may be relevant to how you want to fix it

The RAM guard already computes exactly what the parking path needs (`need N MiB plus M MiB floor, K MiB available, source meminfo`), and it consults `MemAvailable` plus cgroup limits. The checkpoint writer appears to consult nothing. `zram` is not a safety net here: it is compressed RAM, so the spike has nowhere to go, and the copy is a `cudaMemcpy` to host memory that returns `CUDA_ERROR_OUT_OF_MEMORY` rather than an allocator failure the caller could catch earlier.