Pull requests / #3565
#3565 [Engine] serve: auto sessions, an opt-in disk parking around a conversation switch
open · @drumblund · 0 コメント · GitHub で見る
Server & APIAMD / HIPNVIDIA / CUDAModels & quants
本文
<h2>Problem and result</h2><p>The engine has held a saved conversation across restarts for a while: <code>--slot-save-path</code> plus
<code>POST /slots/0?action=save|restore</code> write and read a session file, and <code>SAVE <path></code> / <code>RESTORE <path></code> on
the engine's control stream does the work. What was missing is anything that <em>calls</em> it.</p><p>So a single-slot service had exactly one way to survive a conversation switch: the in-RAM cache, bounded by
<code>--conversation-cache-mib</code>. That bound is easy to hit — at 512K a 27k-token conversation parks 651 MB, so a
24 GiB ceiling holds roughly thirty of them and then starts evicting. And a restart, or a conversation that
was evicted, pays a full re-read of a prompt that can be minutes of prefill.</p><p>After, with <code>"auto_sessions": true</code>:</p>
| before | after
-- | -- | --
switch back to a conversation the RAM cache had dropped | full re-read | restored from its file
27k-token conversation, 512K config | — | saved 27,174 tokens (0.65 GB) in 223 ms, restored in 146 ms
normal service stop | held conversation lost | saved before the stop, restored by the next start
<p>The restored answer matches the one given before the switch character for character.</p><h2>Related discussion</h2><p>none — no issue was opened for this. Off by default (<code>auto_sessions</code> is false unless the config says
otherwise), and every failure is logged and leaves the request alone.</p><h2>Approach</h2><p><strong>The conversation is identified by its first user message.</strong> <code>conversation_key()</code> takes the text of the
first <code>role: user</code> message and returns <code>sha1(text)[:16]</code>, or <code>None</code> when a request has no user text (a
tokenize-only call, say) so it stays out of the way. That is the same boundary the engine's prompt cache
sees, so the two agree on what "a different conversation" means.</p><p><strong><code>_auto_session_turn()</code> runs between requests, holding the service FIFO.</strong> When the incoming request's key
differs from the held one: save the held conversation to <code>auto-<key>.bin</code> (only if it reached
<code>auto_sessions_min_tokens</code>), then restore the incoming one if <code>auto-<key>.bin</code> exists. Same key → nothing
but a token count update. A save that fails does not skip the restore only in one case — a held conversation
with nothing durable, i.e. a cancelled request's provisional state, which the engine refuses as <code>invalid</code>
and which has no file worth protecting.</p><p><strong>A failed step never fails the request.</strong> The whole body is wrapped: <code>print(f"[strata] auto sessions: {e}")</code>
and the request runs as it always did. In particular, if a save failed, the held conversation is still in
the engine and still parks in RAM — restoring over it would replace a full state with a weaker one.</p><p><strong>A normal stop saves what the engine holds</strong> (<code>auto_session_shutdown_save()</code>), but only when the service is
idle: a running request's session is live and saving it would capture a partial state.</p><p><strong>Pruning touches only its own files.</strong> <code>_auto_session_prune()</code> keeps the newest <code>auto_sessions_keep</code> files
and at most <code>auto_sessions_max_gb</code> of them, and deletes only names matching <code>auto-*.bin</code> — a file saved by
hand through <code>/slots/0</code> is never removed.</p><p>Defaults, all overridable from the config: <code>auto_sessions_min_tokens</code> 16000, <code>auto_sessions_keep</code> 12,
<code>auto_sessions_max_gb</code> 100.</p><h2>Validation</h2><p><strong>Passed:</strong></p><p><em>Environment:</em> macOS 26.5.2, M5 Max 128 GB, engine <code>0.1.44.1-metal</code>, IQ3_S 79 GB, 512K context,
<code>--conversation-cache-mib 24576 --conversation-cache-slots 3</code>, <code>slot_save_path</code> set,
<code>auto_sessions_min_tokens</code> 20000.</p><ul><li>A 26,958-token conversation, then a request from a different conversation and back:
<code>auto sessions: saved 27,174 tokens (0.65 GB) to auto-<key>.bin in 223 ms</code>,
<code>auto sessions: restored 27,174 tokens from auto-<key>.bin in 146 ms</code>, answer identical.</li></ul><ul><li>Service stop with a conversation held: saved on shutdown, restored on the next start
(<code>cached=26,987</code> of 26,992 prompt tokens).</li></ul><ul><li><code>python3 -m py_compile serve/server.py</code> clean.</li></ul><ul><li>Restore does not fire for conversations below <code>auto_sessions_min_tokens</code>: logged as skipped, request
unaffected.</li></ul><p><strong>Failed:</strong> none observed.</p><p><strong>Not run:</strong></p><ul><li>A CUDA/HIP/SYCL engine. The change is <code>serve/</code> only and does not depend on the backend, but this machine
has no such hardware.</li></ul><ul><li><code>parallel</code> / <code>--batch</code> setups: <code>/slots/0</code> and this path are refused there by the pre-existing check, since
a session file could interleave with a request being admitted.</li></ul><h2>Limits</h2><ul><li><strong>It is a heuristic.</strong> A client that rewrites the first user message (a system-prompt edit prepended to
the same question, a re-rendered template) starts what looks like a new conversation and gets a new file;
one that keeps the text but changes everything after it reuses the old one. That matches what the prompt
cache does, but it is a guess.</li></ul><ul><li><strong>Disk, not RAM.</strong> The engine's <code>--conversation-cache-mib</code> pool is still the fast path; this is what covers
a restart and a conversation the pool dropped. Both are worth having.</li></ul><ul><li><strong>Depends on the engine's session file.</strong> On a Metal engine before the companion engine fix, <code>SAVE</code> fails
with <code>invalid checkpoint running-state payload</code> — the server logs it and the request proceeds. On CUDA the
path already works.</li></ul><sc-clipboard block-data="[{"blockId":"rcrELWC0VeOIm38Dit4jAN","blockSubtree":{"block":{"rcrELWC0VeOIm38Dit4jAN":{"role":"editor","value":{"id":"rcrELWC0VeOIm38Dit4jAN","parentId":"","type":"header2","props":{"title":[["Problem and result",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"2NAHTbjyACedNb1Zo3Wh7d","blockSubtree":{"block":{"2NAHTbjyACedNb1Zo3Wh7d":{"role":"editor","value":{"id":"2NAHTbjyACedNb1Zo3Wh7d","parentId":"","type":"text","props":{"title":[["The engine has held a saved conversation across restarts for a while: ",null],["--slot-save-path",[["c"]]],[" plus\n",null],["POST /slots/0?action=save|restore",[["c"]]],[" write and read a session file, and ",null],["SAVE <path>",[["c"]]],[" / ",null],["RESTORE <path>",[["c"]]],[" on\nthe engine's control stream does the work. What was missing is anything that ",null],["calls",[["i"]]],[" it.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"5gcnj8ZOi41V2idSUcCs1S","blockSubtree":{"block":{"5gcnj8ZOi41V2idSUcCs1S":{"role":"editor","value":{"id":"5gcnj8ZOi41V2idSUcCs1S","parentId":"","type":"text","props":{"title":[["So a single-slot service had exactly one way to survive a conversation switch: the in-RAM cache, bounded by\n",null],["--conversation-cache-mib",[["c"]]],[". That bound is easy to hit — at 512K a 27k-token conversation parks 651 MB, so a\n24 GiB ceiling holds roughly thirty of them and then starts evicting. And a restart, or a conversation that\nwas evicted, pays a full re-read of a prompt that can be minutes of prefill.",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"WDmTos5sZhTSmNaor990fj","blockSubtree":{"block":{"WDmTos5sZhTSmNaor990fj":{"role":"editor","value":{"id":"WDmTos5sZhTSmNaor990fj","parentId":"","type":"text","props":{"title":[["After, with ",null],["\"auto_sessions\": true",[["c"]]],[":",null]]},"children":[],"parentTable":"block","enabled":true}}}}},{"blockId":"I19oJtibYMLbbh3pcfAjVH","blockSubtree":{"block":{"I19oJtibYMLbbh3pcfAjVH":{"role":"editor","value":{"id":"I19oJtibYMLbbh3pcfAjVH","parentId":"","type":"simple_table","props":{"tableColumnOrder":["Mtkt96b2","Kn8kWurV","aJt1yuCl"],"tableColumnWidthMap":{"Mtkt96b2":120,"Kn8kWurV":120,"aJt1yuCl":120},"tableRowHeader":true},"children":["yOWkqSmlOKEb50hoTnMp0p","AeZAqiusskduslGyNeTset","78tTSMxKLTT17Y5kBqxpUj","OYofUa6yNtyGeOqp194vKZ"],"parentTable":"block","enabled":true}},"yOWkqSmlOKEb50hoTnMp0p":{"role":"editor","value":{"id":"yOWkqSmlOKEb50hoTnMp0p","parentId":"I19oJtibYMLbbh3pcfAjVH","type":"simple_table_row","props":{},"children":["B70Qg2zjOVCwgyvjgCuGai","rCFtb4rzQFDyFiEYb7bFT6","b1tW5yZOfuLypzKEFzGE7r"],"parentTable":"block","enabled":true}},"B70Qg2zjOVCwgyvjgCuGai":{"role":"editor","value":{"id":"B70Qg2zjOVCwgyvjgCuGai","parentId":"yOWkqSmlOKEb50hoTnMp0p","type":"simple_table_cell","props":{"columnKey":"Mtkt96b2"},"children":["PEUeNVYbP9CXtp2FphygsJ"],"parentTable":"block","enabled":true}},"PEUeNVYbP9CXtp2FphygsJ":{"role":"editor","value":{"id":"PEUeNVYbP9CXtp2FphygsJ","parentId":"B70Qg2zjOVCwgyvjgCuGai","type":"text","props":{"title":[]},"children":[],"parentTable":"block","enabled":true}},"rCFtb4rzQFDyFiEYb7bFT6":{"role":"editor","value":{"id":"rCFtb4rzQFDyFiEYb7bFT6","parentId":"yOWkqSmlOKEb50hoTnMp0p","type":"simple_table_cell","props":{"columnKey":"Kn8kWurV"},"children":["MOHs5rkvETGkCQeBNjOKrJ"],"parentTable":"block","enabled":true}},"MOHs5rkvETGkCQeBNjOKrJ":{"role":"editor","value":{"id":"MOHs5rkvETGkCQeBNjOKrJ","parentId":"rCFtb4rzQFDyFiEYb7bFT6","type":"text","props":{"title":[["before",null]]},"children":[],"parentTable":"block","enabled":true}},"b1tW5yZOfuLypzKEFzGE7r":{"role":"editor","value&qu関連リンク
インストール・モデル・リリースへのサイト内リンク。