Issues / #2502
#2502 Metal: a short side request drops the conversation cache (parking is refused, then "invalid checkpoint running-state payload")
open · @joesinvestments · 0 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationLinux
Beschreibung
### What happened On the Metal engine a single short request from another conversation (an agent's "give this chat a title" call) drops the main conversation, and its next turn reads the whole prompt again. Same symptom as #1920 on CUDA, but on a Mac the documented fix (`--conversation-cache-mib`) cannot work, for two reasons we found in v0.1.44 (still the same on `main` 296b59d2): 1. **The memory guard has no macOS reading.** `sample_host_memory()` in `src/core/conversation_memory.cpp` has branches for `_WIN32` and `__linux__` only and returns nothing elsewhere, so every Mac start logs `memory guard: the available RAM cannot be read; parking and session save/restore are refused`. (`src/core/expert_source.cpp` already reads macOS memory with `host_statistics64`.) A small PR for this follows. 2. **With the guard fixed and the cache turned on, parking still fails on every request:** `conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload)`. As far as we can read the code: `conversation_checkpoint_save` / `_restore` have `STRATA_GPU_METAL` branches (`window_checkpoint_save` / `_restore`, `conversation_state.cpp` ~167-194), but the parking path validates with the host layout every time (`view_validate()` ~line 85 and the snapshot check ~line 360 call `conversation_checkpoint_validate()`), while `Window::checkpoint_save` (`metal/src/engine/window.cpp` ~1375-1398) packs the GDN state and history its own way and leaves `block_pos` empty. So the size check cannot pass on Metal. Also: `docs/MAC.md` does not mention the conversation cache, and `metal/setup_mac.py` does not print the `--conversation-cache-mib` tip that `setup.py` prints on a PC. **Repro (one request at a time, greedy, thinking off):** a ~16K-token prompt of real code, two follow-up turns, then a 60-token unrelated request, then a fourth turn of the first conversation. UD-Q4_K_XL streamed from the SSD: | step | prompt tokens | reused | time to first token | |---|---:|---:|---:| | turn 1 | 15,680 | 0 | 22.1 s | | turn 2 | 15,961 | 15,675 | 0.86 s | | turn 3 | 16,239 | 15,956 | 1.32 s | | side request | 60 | 0 | 1.11 s | | turn 4 | 16,519 | **0** | **22.4 s** | Identical with (a) the release engine, (b) the guard fixed, and (c) the guard fixed plus `--conversation-cache-mib 4096 --conversation-cache-slots 4`. IQ3_XXS behaves the same. We have not tried a Metal snapshot ourselves. ### Strata version, GPU, OS Strata v0.1.44 (ed4773eb) built from source with `./setup.sh`, Metal engine. Mac Studio M5 Max (40-core GPU), 64 GB, macOS 27.0.1, `iogpu.wired_limit_mb=58000`. Packs: Unsloth UD-Q4_K_XL and IQ3_XXS (GSQ-RCO), both streaming their experts from the SSD (`STRATA_METAL_STREAM=1`). `--spec 3` with the MTP draft layer, `--prefill auto`, `--max-context 32768`. ### Engine log Release engine: ``` strata generate: memory guard: the available RAM cannot be read; parking and session save/restore are refused strata serve: prompt 15680 tokens = 0 reused + 15680 read in 22122 ms (708.8 tok/s), 256 generated in 6344 ms (40.4 tok/s), drafts accepted 120 of 272, 1 checkpoints strata serve: prompt 15961 tokens = 15675 reused + 286 read in 856 ms (334.2 tok/s), 256 generated in 5527 ms (46.3 tok/s), drafts accepted 140 of 232, 2 checkpoints strata serve: prompt 16239 tokens = 15956 reused + 283 read in 1316 ms (215.1 tok/s), 256 generated in 5562 ms (46.0 tok/s), drafts accepted 140 of 230, 3 checkpoints strata serve: prompt 60 tokens = 0 reused + 60 read in 1112 ms (53.9 tok/s), 16 generated in 282 ms (56.7 tok/s), drafts accepted 11 of 12, 1 checkpoints strata serve: prompt 16519 tokens = 0 reused + 16519 read in 22424 ms (736.7 tok/s), 256 generated in 5348 ms (47.9 tok/s), drafts accepted 137 of 238, 2 checkpoints ``` Guard fixed, `--conversation-cache-mib 4096 --conversation-cache-slots 4`: ``` strata generate: memory guard: limit 64.0 GiB (source: meminfo), current 5.1 GiB, reclaimable cache credited 0.0 GiB; available 58.9 GiB (MemAvailable 58.9 GiB), floor 2560 MiB strata serve: conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload) strata serve: prompt 60 tokens = 0 reused + 60 read in 1150 ms (52.2 tok/s), 16 generated in 281 ms (56.9 tok/s), drafts accepted 11 of 12, 1 checkpoints strata serve: conversation cache: skip parking (conversation snapshot: invalid checkpoint running-state payload) strata serve: prompt 16519 tokens = 0 reused + 16519 read in 22374 ms (738.3 tok/s), 256 generated in 5921 ms (43.2 tok/s), drafts accepted 137 of 238, 2 checkpoints ```
Mehr auf der Site
Links zu Install, Modellen, Releases.