#5895 [Engine] Long agent sessions on 64 GB RAM: a 260k conversation can't be parked, so every side request forces a full reread (follow-up to #57 / #190)

open · @SlowSoapSoak · 8 comments · View on GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

### Setup

- Strata 0.2.13, engine 0.1.59 (Windows 11, CUDA 13), IQ3_S, `--max-context 393216`, `--kv int8`
- RTX 5090 32 GB + RTX 3090 24 GB (layer split), 64 GB RAM, 16 cores / 32 threads
- Client: Claude Code 2.1.321 in Claude Desktop (third-party gateway, Messages API)

### What happens

Before each built-in browser tool call, Claude Code sends a side request: its automatic safety check, the auto-mode classifier. This happens outside auto mode too. With auto mode on, Bash calls trigger it as well. The side request has its own system prompt (always the same 64,829-token prefix), ~70k tokens in total, and `max_tokens 2112`. That is the A → B → A pattern from #41 / #57, where A is a long agent session:

```
prompt 258080 tokens = 257877 reused +    203 read in  1029 ms   # A continues
prompt  71155 tokens =      0 reused +  71155 read in 23709 ms   # B: safety check
prompt 258960 tokens =      0 reused + 258960 read in 83622 ms   # back to A
```

Each browser step costs ~105 s of prompt reading instead of about 1 s.

### The conversation cache can't park A on this machine

With `--conversation-cache-mib 8192`:

```
conversation cache: skip parking (physical RAM admission; need 4595 MiB plus 2560 MiB floor, 2782 MiB available, source meminfo; evicted 1, 0 still parked)
```

During the session, MemAvailable stayed between ~0.6 and ~4.5 GiB. The expert arena locks 40,290 MiB, and the engine's working set is ~51 GiB of 63 GiB. B (~1.8 GiB snapshot) parked and came back fine once (`restored 67901 tokens (checkpoint) in 201.8 ms`). A (260k tokens, 4.3–4.6 GiB snapshot) never fit. Lowering `--conversation-cache-min-free-mib` can't close a gap that large.

`--batch 2` (batch-groups auto: 2 groups of 1 slot) doesn't help either. Both slot sessions are sized for the full context (6.60 GiB each on CUDA1), so the 5090's expert cache dropped from 9,882 to 815 slots, and prompt reading fell from ~3,100–3,500 to ~1,060 tok/s. The one return to A still began at `0 reused` (one data point: I cancelled the run after that).

### Questions

1. Is the NVMe tier from #190 still planned? Writing and reading a 4.5 GiB snapshot on NVMe would take seconds, not an 80 s reread, and it removes the RAM admission limit on 64 GB machines.
2. On multi-GPU setups, the arena keeps a RAM copy of every expert, including the ~14k that sit on the cards. Could there be an option to keep host copies only for non-resident experts, plus the slots the prompt path borrows? The freed RAM would be enough to park A.
3. Or a small second slot for short side requests (say up to 96k tokens), so the long conversation's slot is never overwritten. `--batch 2` comes close, but it sizes both slots for the full context.

I can run test builds and share full logs.

Related on this site

Quick links to related pages.