Pull requests / #3035

#3035 docs: explain cache settings for long-context batch handoffs

open · @oscarlius · 0 コメント · GitHub で見る

BenchmarksSetup & installModels & quantsDocumentation

本文

## Summary

Document why a long request that starts alone can reread its prompt when it is moved into a batch slot with `--prompt-cache 0`. Add a minimal configuration that keeps the live handoff state and explain how to verify the reused/read token counts.

## What changed

Only `docs/BATCHING.md` changes. The new section explains:

- `--prompt-cache 1 --prompt-cache-every 0 --prompt-cache-root 0` and its per-cache-set/slot memory cost.
- The separate parked-conversation budget: the minimal example has parking off, but users with parking enabled should retain their existing budget.
- Why in-flight `running` requests do not establish the number of actively decoding slots, and why this configuration does not enable parallel cold-prompt prefill.
- Checking the engine's handoff log and actual slot/status metrics without claiming that all configured sessions remain resident.

No defaults, source code, deployment files or live service settings change. This documentation PR is independent of the cache-observer fix in #3034.

## Validation

- Compared the guidance with the current engine's exact-prefix/live-state reuse conditions, slot-cache setup, checkpoint options and server handoff path.
- Checked all four example flags against the current engine and parsed the example argument fragment as JSON.
- `git diff --check` passes; the diff contains only this documentation section.
- No new GPU run or service restart. No benchmark improvement is claimed in the document.

関連リンク

インストール・モデル・リリースへの站内リンク。