#4374 [Other] serve: opt-in "system_reminders": "tail" keeps a client's system-reminder blocks at the end of the prompt

open · draft · @XinacS · 0 评论 · 去 GitHub 看

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

说明

**Draft on purpose: the behaviour is implemented and unit-tested, the hardware measurement is running now. No performance claim in this PR yet.**

## Problem and result

Agents inject a `<system-reminder>` block into a conversation turn (opencode's plan/build mode change, Claude Code's environment notes) and drop it again on the next request. A conversation slot holds the prompt's tokens *plus* what the engine generated after them, so the held prefix stops matching at that block, the restore fails, and tokens the model already processed are read again.

Recorded on this PC (see environment below) between 10 and 27 Oct 2026, from the diagnostic this PR adds: **991** requests whose divergence point names a `<system-reminder>` block. Re-read from nothing per request: median **72,609** tokens, maximum **211,971**, total **79,851,796**. Worst day (25 Oct): a 220,166-token prompt kept a shared prefix of 8,195 tokens.

With the opt-in `"system_reminders": "tail"`, every reminder block is lifted out of the earlier turns onto the prompt's last user turn, so what the client sends in the earlier turns stays byte-stable and the block is the newest thing the model sees.

## Related discussion

Same class of problem as the pinned billing stamp (`pin_billing_stamp` in `serve/frontend.py`) and as the tool-order fix (#1624, landed in 0.1.42 as the first-seen tool order): what changes inside the history costs the whole conversation.

Client side, for context (all closed, none fixed): opencode [anomalyco/opencode#23595](https://github.com/anomalyco/opencode/issues/23595) (closed `not_planned` 2026-10-03), [#58485](https://github.com/anomalyco/opencode/issues/58485) (closed `not_planned` 2026-10-26), open PRs [#57065](https://github.com/anomalyco/opencode/pull/57065) and [#58395](https://github.com/anomalyco/opencode/pull/58395). Claude Code reports the same effect (anthropics/claude-code#48734).

## Approach

`serve/frontend.py` gains `tail_system_reminders(messages)`: it takes every `<system-reminder>...</system-reminder>` block out of the turns before the last user turn and appends them, in the order the client sent them, to the end of that turn. A block already in the tail is not copied again, and the tail is built per turn so a block is never duplicated. It runs after `_env_to_user`, so the `<env>` block stays at the front of the first user turn and the reminder blocks land at the end of the last.

It is reached through a keyword on the 2 converters (`openai_to_messages`, `anthropic_to_messages`) and through `responses.input_messages`, because `/v1/responses` (#451) builds its messages in `serve/responses.py` and never runs the chat converter - Codex CLI, a client that switches modes, goes there.

`"system_reminders"` accepts `"keep"` (default) or `"tail"`; any other value stops the server at start with the bad value named. **With the default, every request renders byte-for-byte as in 0.2.7: the code path is inert.** No engine code is touched, so no CUDA/HIP/SYCL build is involved.

The PR also adds `Service.note_divergence`, a measurement aid: for each batch slot it compares the request's token ids with the tokens that slot holds for its next turn, and when a slot's held prefix is not fully shared it appends one line to `<log>.divergence` (prompt length, slot, held length, shared prefix, 24 tokens decoded around the cut, last cache event). It writes only to that sidecar file and changes nothing the engine reuses. Without it the numbers above were invisible.

## Validation

Environment for everything below: Linux, RTX PRO 5000 Blackwell (sm_120), Swift 1.5 IQ3_XXS pack, 262,144-token context, `--conversation-cache-slots 16`, `--batch 3`, `--prompt-cache-tail`, engine 0.1.52, server revision `649a5bd7` on `main` = `5e8a4cca` (0.2.7).

**Passed** (no GPU: the serve tests use the mock engine)

- `.venv/bin/python -m unittest serve.test_server -k Reminder -v` → 4 tests OK: a block in a past turn moves to the last turn and the past turn becomes byte-stable; a block already in the last turn is not moved twice; 2 blocks in 2 past turns are moved once each; the `/v1/responses` route moves the block too.
- `.venv/bin/python -m unittest serve.test_server -k Divergence -v` → 2 tests OK (a replayed reply that cuts the held prefix is logged; a request that only extends it is not).
- `.venv/bin/python -m unittest serve.test_server` → OK.
- `.venv/bin/python -m unittest serve.test_responses` → OK (1 skipped).
- Whole `serve/` surface, 72 test files, run side by side on this revision and on clean `5e8a4cca`: identical results.

**Failed**

- `.venv/bin/python -m unittest serve.test_keepawake` → `test_service_holds_for_the_whole_request_including_early_close` fails. It fails on clean 0.2.7 without this change on the same machine, so it is pre-existing here (Windows `SetThreadExecutionState` behaviour), not caused by this PR.

**Not run** (this is why the PR is a draft)

- Live A/B of `"keep"` vs `"tail"` on an agent session that switches plan/build: reused and read tokens and prompt-read milliseconds from `GET /metrics` and the engine's `prompt N tokens = X reused + Y read in Z ms` line; planned as arms `keep`/`tail`, order `ABBA`, 3 repetitions, with `tools/agent_turns_bench.py`.
- Reply token-id parity between arms (`tools/conversation_cache_parity.py`).
- Agent task pass rate, so that a cache win is not paid for in answers.
- Byte parity of rendered prompts over captured real requests with the switch off.
- Windows, Mac, AMD/HIP, multi-GPU: not tested (the change is Python in `serve/`, no backend code).

## Limits

- Only `<system-reminder>` blocks are moved, and only in turns whose content is a plain string; a turn with image/list content is left alone.
- The last user turn is rewritten, so the cut moves to the end of that turn: everything after it (the current turn's generation) is still read. The engine's opt-in `STRATA_CACHE_MESSAGE_BOUNDARY` (docs/MESSAGE_BOUNDARY_CACHE.md) is the counterpart for that case; measuring the two together is part of the test in progress.
- **Open question, under test:** a turn that carried a block becomes a past turn on the next request, and moving its block out rewrites a turn a slot may already hold. That can put the divergence point back into the history - the 25 Oct rows above (prefix 8,195 of 220,166) are consistent with it. I will report the measured A/B either way, including a null result.
- `.strip()` of a cleared turn changes its trailing whitespace: the cut in today's log sits exactly at `'…a fix<|im_end|>'` vs `'…a fix\n<system-reminder>'`. Whether keeping the client's whitespace is better is measured, not assumed.
- `note_divergence` compares the request's tokens with each slot's held tokens in Python for every request of 4,096 tokens or more. Measured cost of the comparison loop alone, worst case (85% shared prefix): 0.46 ms at 8,192 tokens with batch 3, 1.85 ms at 32,768, 6.46 ms at 131,072, 12.65 ms at 262,144. Say the word if you want it gated behind the same switch so the default path does no extra work; I would rather gate it.

本站相关内容

相关页面的快捷入口。