Issues / #3090

#3090 The prompt-cache pin protects the oldest checkpoint, which is not the system-prompt root when the system prompt is longer than --prompt-cache-every

open · @mikhashev · 0 commentaires · Sur GitHub

BenchmarksServer & APIModels & quantsWindows

Description

> Strata 0.1.45 (`--serve`, Windows), Qwen3.8-Flash-Next IQ3_XXS, default `--prompt-cache 6`, `--prompt-cache-every 16384`, `--prompt-cache-root 2048`. Read from the 0.1.45 sources and observed in our engine log.
>
> **What the code intends.** `conv_cache.hpp:12-24` describes the chain's root as "in practice the end of the system prompt", pinned so that every new chat of the same client mounts through it. The root checkpoint is placed at the first turn boundary at or after `--prompt-cache-root` on a read from token 0 (`generate.cpp:13313-13325`).
>
> **What the code does.** `eviction_victim` pins index 0, the oldest chain item (`conv_cache.hpp:35-41`). On a read from token 0, the periodic checkpoints are taken during the read (`on_chunk`, `generate.cpp:9265-9325`), before the read reaches the system-prompt end where the root is saved. So when the system prompt is longer than `--prompt-cache-every`, index 0 is the first grid checkpoint (16,384), the pin protects it, and the real root is an ordinary entry that rotates out under least-recent-use.
>
> **Observed.** Our agent client's system message is longer than 16,384 tokens. Over a long conversation, 7 of 7 turn boundaries resumed from exactly 16,384 tokens and re-read 76K–119K tokens (21–35 s each at ~3,500 tok/s). `--prompt-cache-spread` fixed the turn-start case for us (the turn-start checkpoint now survives the turn), so this is not blocking — but the pin still guards a grid point rather than the shared prefix it was written for, and a new chat or a session reset with the same system prompt falls back to 16,384.
>
> **Request.** Pin the checkpoint saved at the system-prompt root explicitly (for example by marking it when it is saved), rather than by position in the chain — or, if the current behaviour is intended, adjust the comment at `conv_cache.hpp:12-24`, which assumes index 0 is the root.
>
> Related: #3088 (same deployment).

Sur le site

Liens install, modèles, releases.