#6276 [Engine] Re-land the conversation-cache disk-spill tier (closed #5399) with the disk-restore erase-order fix

open · @satwareAG-ironMike · 0 comentários · No GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurity

Descrição

### Why this issue exists

PR #5399 (the conversation-cache disk-spill tier) was closed by the housekeeping bot on 2026-11-12 after the format warning. The format was fixed on 2026-11-15 and `/reopen` was attempted, but GitHub would not reopen it (Validation Failed - the branch appears deleted), so per the bot's own remedy this is the re-land request: **open a new pull request from a restored branch and link #5399.** Nothing here is a rejection of the content - the content is proven in production.

### What the tier does (from #5399, unmodified description)

When the parked RAM conversation cache fills, the oldest parked conversation spills to `*.conv` files instead of being dropped, and a returning conversation is read whole back off the disk; the RAM shell keeps ids/images so matching keeps working. Off by default (`--conversation-cache-file-mib 0` = current behavior). The new `conversation_cache_disk_test` covered the byte round-trip under ASan/UBSan.

### Production evidence (all from the #5399 thread)

- nrtdadmin, 2026-11-10, 4x V100 box, rebased onto v0.2.13 with monitor counters (`d272ca9f4` + the `f027b79f9` eviction-order fix + `6ecf98cd9` counters): a full loaded cycle with the RAM cache pinned at 127/128 GiB and 469 parked conversations wrote **317 spill files and served 8 whole-conversation restores in one day**, `evictions` down to **25** - before the fix, the same pressure produced 334 evictions and at most ONE file on disk.
- satwareAG-ironMike, 2026-11-10, single-GPU box (RTX 3060 12 GB, 62 GiB RAM, engine 0.1.56 / serve v0.2.12, Qwen3.8-Flash-Next IQ3_XXS at 196K context): the boot-auto budget (5376 MiB) fits exactly ONE 130-146K-token conversation snapshot (3.0-3.5 GB each), so two alternating long agent sessions evict each other at every switch and the loser re-reads the whole prompt cold - 130-190 s per switch. This is the demand side for the tier outside many-GPU hosts.

### A must-fix for any re-land: the disk-restore erase order (found by keithm999, 2026-11-11)

On v0.2.13 (HIP build, R9700 / gfx1201) every restore from disk failed: in `take()`, `erase(index)` runs BEFORE the file is read, and `erase()` deletes the file, so `conversation_cache_disk_read()` always fails with "cannot open". Moving `erase(index)` to after the read fixed it (their box: 19 of 22 returning sessions came back from disk, median time-to-first-token 9.9 s vs 84.7 s without it). Two asks for the re-land:

1. include the erase-after-read order, and
2. add a test that would have caught it (a `take()`-level round-trip through `conversation_cache_disk_read()`, not only the byte-level helper) - the current round-trip test passed while every real restore failed, and `++disk_reads_` also counts failed reads, which hid it behind a green counter.

### Ask

@nrtdadmin (author of record) - please re-land the three commits as a new PR (branch restored or re-created), format-compliant, incorporating the erase-order fix and the stronger test. Happy to validate on the 3060/62 GiB box above with our guarded A/B window discipline before merge.

No site

Links install, modelos, releases.