#6658 [Engine] SSD conversation tier permanently deletes a snapshot when the read-back RAM gate refuses it

open · @xvcezrqpbxtpzblrt · 0 comentários · No GitHub

Server & APIModels & quantsWindows

Descrição

Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.

**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.

> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.

### Area


[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?


Source (the `strata serve` conversation cache)

### What happened
The conversation cache's SSD tier is designed to keep a parked conversation when RAM cannot hold it: an evicted image is written to a session file and a later request whose prompt matches it reads the prefix back instead of re-reading the whole prompt. The read-back path, however, applies the physical-RAM admission gate **before** reading, and treats a gate refusal exactly like a corrupt or foreign file: it removes the entry from the in-memory list and deletes its file. The next request that needs that conversation can no longer find it in the SSD tier and re-reads the entire prompt from token 0.

This is a different failure from a park that was never captured: here the snapshot exists on disk and is a perfect prefix match, but the engine destroys it at the moment it is most needed — when RAM is tight. A transient, recoverable condition (not enough free RAM at this instant) destroys a durable asset (a multi-gigabyte K/V snapshot and the prefill it represents), with no retry.

### Smallest steps or command that reproduce it
Not reproduced locally. The suspected path, from source only:

1. Run `strata serve` with the conversation cache enabled and the SSD spill tier at its default (the tier is on when `--conversation-spill-mib` is unset; it is `-1` by default and only `0` disables it).
2. Drive a long single conversation until a parked image is evicted from RAM and written to the SSD tier. The log line is `strata serve: conversation cache: spilled <N> tokens to the SSD tier (evicted from RAM, written in the background) ...`.
3. Reach a state where free RAM is below `--conversation-cache-min-free-mib` plus the snapshot's read peak, and send a request that continues that same conversation (so the SSD entry is the best prefix match).
4. The suspected result: `strata serve: conversation cache: could not read <N> tokens back from the SSD tier (not enough RAM to read it ...); the prompt is read again`, followed by a full re-prefill. A subsequent request with the same prefix no longer finds the entry in the SSD tier.

What would verify it: instrument `ConversationSpill::take_serial` to log the `SessionStatus` passed by `session_file_read` and the `SessionError::memory` case specifically, then confirm that the file is deleted on that branch and that `spill.best()` no longer matches that conversation on the next request. No runtime verification was performed for this draft.

### Reproduction status
Unconfirmed static analysis (I read code or ran a scanner; I did not see it fail)
### Strata version

0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)

### Graphics card, memory and system
Not recorded; no local run was performed. The code path is in the host-side cache and is not specific to a GPU vendor.

### Model, size and settings
Any `--serve` configuration with a conversation cache and the SSD spill tier enabled (default). The effect grows with snapshot size: long contexts and large K/V make the lost prefill expensive, and the SSD spill cost rule already selects those long conversations for the tier.

### Logs
No runtime log is available for this draft. Verbatim source evidence is given in the next section.

### Why this is worth fixing for everyone
The SSD tier is on by default for every `--serve` user with a conversation cache; it is not an opt-in experiment. Its entire purpose is to preserve a conversation exactly when RAM is under pressure. But the read-back path refuses to use it under that same condition, and then deletes it. The users most affected are precisely those the tier was built for: long agent sessions, large contexts, and machines where free RAM dips near the floor at read time. A single transient dip costs a multi-gigabyte, multi-minute prefill that cannot be recovered, and the loss is permanent rather than deferred. Fixing this makes the default tier actually deliver its intended benefit to all users without changing any setting or hardware.

### Root cause (source evidence)

1. The SSD tier is enabled by default; only `0` turns it off.

`engine/src/program/generate.cpp:1105`:

```cpp
    int64_t conversation_spill_mib = -1;
```

`engine/src/program/generate.cpp:13542`:

```cpp
        if (conversations.enabled() && stages.empty() && o.conversation_spill_mib != 0) {
```

2. The read-back path gates the read on current free RAM, and on failure both drops the in-memory entry and deletes the file.

`engine/src/program/generate.cpp:18798-18819`:

```cpp
                        strata::core::SessionReadLimits limits;
                        const uint64_t floor = (uint64_t) o.conversation_cache_min_free_mib << 20;
                        pools_budget.shrink((size_t) -1);
                        limits.admit = [floor, &o](uint64_t need, std::string& w) {
                            const auto mem = strata::core::sample_host_memory();
                            if (strata::core::conversation_memory_admit(mem ? std::optional<uint64_t>(mem->available) : std::nullopt,
                                                                        need, floor)) return true;
                            w = "not enough RAM to read it (" + std::to_string(need >> 20) + " MiB plus a floor of " +
                                 std::to_string((long long) o.conversation_cache_min_free_mib) + " MiB needed, " + ram_note(mem) + ")";
                            return false;
                        };
                        if (strata::core::conversation_session_read_limits(limits, ss, g, use_mtp ? &mtp.kv_state() : nullptr,
                                (uint64_t) o.max_context, (uint64_t) std::max(o.prompt_cache, 1), why) &&
                            spill.take_serial(sp.serial, *spill_id, limits, image, why))
                            from_ssd = true;
                        else
                            spill.drop_serial(sp.serial);   // (limits that cannot be formed: the entry cannot be read here either)
```

`engine/src/program/generate.cpp:18829-18832`:

```cpp
                    } else {
                        std::fprintf(stderr, "strata serve: conversation cache: could not read %zu tokens back from the SSD tier "
                                             "(%s); the prompt is read again\n", sp_tokens, why.c_str());
                    }
```

3. `ConversationSpill::take_serial` removes the entry from the list before the read and deletes the file unconditionally afterwards, regardless of why the read failed. It does not pass a `SessionStatus` to `session_file_read`, so it cannot tell a memory refusal from a bad file.

`engine/include/strata/core/conversation_spill.hpp:320-339`:

```cpp
    /// Reads entry `serial` back into `out` (bounded by `limits`, the identity checked) and removes the entry. False: `out` is
    /// unchanged and the entry is gone (or was not there).  The file is read outside the lock: the entry leaves the list first.
    bool take_serial(uint64_t serial, const SessionFileIdentity& id, const SessionReadLimits& limits, SavedConversation& out,
                     std::string& error) {
        std::filesystem::path f;
        {
            std::lock_guard<std::mutex> lk(mu_);
            const size_t i = index_locked(serial);
            if (i == SIZE_MAX) { error = "no such SSD entry"; return false; }
            f = entries_[i].file;
            budget_.drop(entries_[i].bytes);
            entries_.erase(entries_.begin() + (std::ptrdiff_t) i);
        }
        SavedConversation image;
        size_t read = 0;
        const bool ok = session_file_read(f.string(), id, image, read, error, limits);
        std::error_code ec;
        std::filesystem::remove(f, ec);
        if (ok) out = std::move(image);
        return ok;
    }
```

The design comment states the same policy explicitly.

`engine/include/strata/core/conversation_spill.hpp:226-228`:

```cpp
/// budget, and never leaves a partial file behind; take() reads an entry back under the caller's read limits and removes it either way (a
/// file that does not read back is not tried again).
```

4. The file reader already distinguishes a memory refusal from a corrupt/foreign file, so the information needed to keep the entry exists but is discarded by the spill layer.

`engine/src/core/conversation_file.cpp:1216-1222`:

```cpp
    if (limits.admit) {
        std::string why;
        if (!limits.admit(session_read_peak_bytes(size), why)) {
            st.error = SessionError::memory;
            error = "session file: " + why;
            return false;
        }
    }
```

### Proposed fix
Keep the deletion policy only for permanent failures (bad header, checksum, wrong model/config fingerprint, size mismatch). For `SessionError::memory` — and for any other transient read failure — leave the entry and its file in the SSD tier, restore its budget accounting, and let a later request retry; the read gate is re-evaluated on every request, so a transient dip resolves on its own. A small guarded retry budget (or a `next_retry` timestamp) can prevent a permanently tight machine from retrying the same entry on every turn. The same distinction should be applied at the `generate.cpp:18815` call site, which currently deletes the entry whenever the read limits cannot be formed. This is a local change to the spill read path and does not alter capture, eviction, or the RAM gate itself.

### Duplicate search

Searched open+closed issues for `SSD tier`, `spill`, `conversation cache read back`, `disk spill`. Closest, none a duplicate: #6276 (open, re-land the disk-spill tier PR), #6281 (open, SSD tier turns off when MTP comes from a GGUF — a fingerprint bug), #3036 (open, parked snapshot copies corrupt on Windows/WDDM), #5895 and #6655 (open, snapshot refused at *park* time — the other memory gate, not delete-on-read). None covers the take-back path deleting an otherwise valid file when the read RAM gate refuses it.

No site

Links install, modelos, releases.