Issues / #2502

#2502 Metal: a short side request drops the conversation cache (parking is refused, then "invalid checkpoint running-state payload")

open · @joesinvestments · 0 comentários · No GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationLinux

Descrição

### What happened

On the Metal engine a single short request from another conversation (an agent's "give this chat a title" call) drops the
main conversation, and its next turn reads the whole prompt again. Same symptom as #1920 on CUDA, but on a Mac the
documented fix (`--conversation-cache-mib`) cannot work, for two reasons we found in v0.1.44 (still the same on `main`
296b59d2):

1. **The memory guard has no macOS reading.** `sample_host_memory()` in `src/core/conversation_memory.cpp` has branches
   for `_WIN32` and `__linux__` only and returns nothing elsewhere, so every Mac start logs
   `memory guard: the available RAM cannot be read; parking and session save/restore are refused`.
   (`src/core/expert_source.cpp` already reads macOS memory with `host_statistics64`.) A small PR for this follows.
2. **With the guard fixed and the cache turned on, parking still fails on every request:**
   `conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload)`.
   As far as we can read the code: `conversation_checkpoint_save` / `_restore` have `STRATA_GPU_METAL` branches
   (`window_checkpoint_save` / `_restore`, `conversation_state.cpp` ~167-194), but the parking path validates with the
   host layout every time (`view_validate()` ~line 85 and the snapshot check ~line 360 call
   `conversation_checkpoint_validate()`), while `Window::checkpoint_save` (`metal/src/engine/window.cpp` ~1375-1398)
   packs the GDN state and history its own way and leaves `block_pos` empty. So the size check cannot pass on Metal.

Also: `docs/MAC.md` does not mention the conversation cache, and `metal/setup_mac.py` does not print the
`--conversation-cache-mib` tip that `setup.py` prints on a PC.

**Repro (one request at a time, greedy, thinking off):** a ~16K-token prompt of real code, two follow-up turns, then a
60-token unrelated request, then a fourth turn of the first conversation. UD-Q4_K_XL streamed from the SSD:

| step | prompt tokens | reused | time to first token |
|---|---:|---:|---:|
| turn 1 | 15,680 | 0 | 22.1 s |
| turn 2 | 15,961 | 15,675 | 0.86 s |
| turn 3 | 16,239 | 15,956 | 1.32 s |
| side request | 60 | 0 | 1.11 s |
| turn 4 | 16,519 | **0** | **22.4 s** |

Identical with (a) the release engine, (b) the guard fixed, and (c) the guard fixed plus `--conversation-cache-mib 4096
--conversation-cache-slots 4`. IQ3_XXS behaves the same. We have not tried a Metal snapshot ourselves.

### Strata version, GPU, OS

Strata v0.1.44 (ed4773eb) built from source with `./setup.sh`, Metal engine. Mac Studio M5 Max (40-core GPU), 64 GB,
macOS 27.0.1, `iogpu.wired_limit_mb=58000`. Packs: Unsloth UD-Q4_K_XL and IQ3_XXS (GSQ-RCO), both streaming their experts
from the SSD (`STRATA_METAL_STREAM=1`). `--spec 3` with the MTP draft layer, `--prefill auto`, `--max-context 32768`.

### Engine log

Release engine:
```
strata generate: memory guard: the available RAM cannot be read; parking and session save/restore are refused
strata serve: prompt 15680 tokens = 0 reused + 15680 read in 22122 ms (708.8 tok/s), 256 generated in 6344 ms (40.4 tok/s), drafts accepted 120 of 272, 1 checkpoints
strata serve: prompt 15961 tokens = 15675 reused + 286 read in 856 ms (334.2 tok/s), 256 generated in 5527 ms (46.3 tok/s), drafts accepted 140 of 232, 2 checkpoints
strata serve: prompt 16239 tokens = 15956 reused + 283 read in 1316 ms (215.1 tok/s), 256 generated in 5562 ms (46.0 tok/s), drafts accepted 140 of 230, 3 checkpoints
strata serve: prompt 60 tokens = 0 reused + 60 read in 1112 ms (53.9 tok/s), 16 generated in 282 ms (56.7 tok/s), drafts accepted 11 of 12, 1 checkpoints
strata serve: prompt 16519 tokens = 0 reused + 16519 read in 22424 ms (736.7 tok/s), 256 generated in 5348 ms (47.9 tok/s), drafts accepted 137 of 238, 2 checkpoints
```
Guard fixed, `--conversation-cache-mib 4096 --conversation-cache-slots 4`:
```
strata generate: memory guard: limit 64.0 GiB (source: meminfo), current 5.1 GiB, reclaimable cache credited 0.0 GiB; available 58.9 GiB (MemAvailable 58.9 GiB), floor 2560 MiB
strata serve: conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload)
strata serve: prompt 60 tokens = 0 reused + 60 read in 1150 ms (52.2 tok/s), 16 generated in 281 ms (56.9 tok/s), drafts accepted 11 of 12, 1 checkpoints
strata serve: conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload)
strata serve: prompt 16519 tokens = 0 reused + 16519 read in 22374 ms (738.3 tok/s), 256 generated in 5921 ms (43.2 tok/s), drafts accepted 137 of 238, 2 checkpoints
```

No site

Links install, modelos, releases.