Issues / #2714
#2714 [Feature]: let a coding agent's own heavy jobs have the GPU and RAM, then continue the conversation without re-reading it
open · @page1 · 0 Kommentare · Auf GitHub
NVIDIA / CUDAModels & quantsWindows
Beschreibung
### What do you want to do?
I run coding agents (opencode, Claude Code) against Strata on one PC. Their work often includes running a program that needs the same GPU and RAM the model holds: a simulation, a training run, an ML pipeline. What I want:
1. **The model gets out of the way while the job runs.** It gives back the RAM and the VRAM, and stays out: nothing loads it back over the job, not even another agent's request. With several agents running jobs, it stays out until the last job ends, however each one ends (a crash or a kill included).
2. **The model comes back and continues the same conversation efficiently.** The agent then reads the results and carries on where it was. These conversations are long (100K-250K tokens with tool output), and today a restart reads each one again from token 0.
Measured on an RTX 3090 24 GB, Core Ultra 9 285K, 95 GB DDR5-6800, Windows 11, Flash-Next IQ3_S at 262K context, with a ~100K-token conversation, timed from the moment the model is asked back to the first answer of the continued conversation:
| | Back with the conversation |
|---|---|
| today (start the engine, then read the whole prompt again) | 53.2-53.8 s |
| #1937's disk tier + the latch (`POST /load`, the conversation from the disk) | 18.1-18.7 s |
| + the start-up PRs (#2672, #2673) and other local start changes | **12.9-13.2 s** |
Giving the machine to the job: 4.2-4.5 s. In use on the whole PC before -> after: RAM 81.2 -> 28.0 GB (of 95.5; the rest is other programs), VRAM 23.2 -> 1.0 GB (of 24).
**Why a fast start matters for this, and not much otherwise.** For a server that stays up all day, the engine start happens about once a day, and a few seconds there are worth little. In this workflow the start is part of every iteration: the agent hands the whole machine to a job and takes it back, many times a day, and every return waits for a full engine start. Once the conversation itself comes back from the disk in a second or two, the start is most of what is left: here about 11 s of the 13 s (`POST /load` returned after 10.9-11.0 s), and the engine start went from 17.0 s on v0.1.44.1 to 11.9 s (median of 6 starts each).
How it's built locally:
- the persistent disk tier of #1937, with a small fix for reading back under tight RAM: xrip/Strata#1, a PR against #1937's branch (the conversation comes back in 2.4 s instead of being read again in 39 s when free RAM is close to the floor);
- a 503 "latch": `POST /unload {"save": true}` saves and ends the engine, and requests answer 503 until `POST /load` (xrip/Strata#2, a PR against #1937's branch);
- "holds": process ids that keep the model out until they end;
- a CLI and opencode commands to wrap a job.
The latch is open as xrip/Strata#2 (server-side, about 115 lines with 9 tests); holds would follow on top of it. If #1937 does not land, the same can be built standalone on the session files. I'd like to know whether you want this in Strata, and in which shape.
Mehr auf der Site
Links zu Install, Modellen, Releases.