Pull requests / #3334

#3334 [Docs] Explain cache settings for long-context batch handoffs

open · @oscarlius · 0 commentaires · Sur GitHub

Models & quantsDocumentationWindows

Description

## Problem and result

The batching guide still omits the cache settings needed to reuse a solo request's live prefix when it moves into a batch slot. Add a minimal handoff configuration, separate its memory cost from parked-conversation budgets, and explain why in-flight requests do not establish concurrent cold-prompt prefill.

## Related discussion

Continues #3035 after the 0.2.0-beta tracker reset. The original passed the new format check, but GitHub refused to reopen the rebased branch; [the reset bot](https://github.com/Niko1221/Strata/pull/3035#issuecomment-6282591198) explicitly suggested a new linked pull request.

None required for this documentation clarification. This updates the original PR after the 0.2.0-beta tracker reset.

## Approach

Only `docs/BATCHING.md` changes. Document `--prompt-cache 1 --prompt-cache-every 0 --prompt-cache-root 0`; keep an existing conversation-parking budget when already enabled. Explain observed slot/status checks without claiming all configured slots remain resident.

## Validation

**Passed:**
- Rebased on upstream main `e88ee39d1d6be6e3364183af3b39b62a7ed6f211` (0.2.0-beta), candidate `91e5a5d5b6e52c5b4951655ab7f66f2627de6ec7`. Windows, Python 3.12.7.
- Confirmed this guidance is absent from the base document and appears once in the candidate.
- Parsed the example argument fragment as JSON after removing its explicit ellipsis placeholder; all four flags exist in the current native argument parser.
- Checked the current `o.prompt_cache > 0` live-prefix reuse gate and per-slot cached-state gates in `src/program/generate.cpp`; the prose matches them.
- Checked new-section relative links and `git diff --check e88ee39d1d6be6e3364183af3b39b62a7ed6f211...HEAD`; only `docs/BATCHING.md` differs.

**Failed:**
- None observed in these documentation checks.

**Not run:**
- A new native handoff generation, model/GPU test or backend build. No runtime source or default changes, and no speedup is claimed.

## Limits

This is documentation and source-consistency validation, not measured handoff or hardware-capacity evidence.

Sur le site

Liens install, modèles, releases.