Pull requests / #3334
#3334 [Docs] Explain cache settings for long-context batch handoffs
open · @oscarlius · 0 commentaires · Sur GitHub
Models & quantsDocumentationWindows
Description
## Problem and result The batching guide still omits the cache settings needed to reuse a solo request's live prefix when it moves into a batch slot. Add a minimal handoff configuration, separate its memory cost from parked-conversation budgets, and explain why in-flight requests do not establish concurrent cold-prompt prefill. ## Related discussion Continues #3035 after the 0.2.0-beta tracker reset. The original passed the new format check, but GitHub refused to reopen the rebased branch; [the reset bot](https://github.com/Niko1221/Strata/pull/3035#issuecomment-6282591198) explicitly suggested a new linked pull request. None required for this documentation clarification. This updates the original PR after the 0.2.0-beta tracker reset. ## Approach Only `docs/BATCHING.md` changes. Document `--prompt-cache 1 --prompt-cache-every 0 --prompt-cache-root 0`; keep an existing conversation-parking budget when already enabled. Explain observed slot/status checks without claiming all configured slots remain resident. ## Validation **Passed:** - Rebased on upstream main `e88ee39d1d6be6e3364183af3b39b62a7ed6f211` (0.2.0-beta), candidate `91e5a5d5b6e52c5b4951655ab7f66f2627de6ec7`. Windows, Python 3.12.7. - Confirmed this guidance is absent from the base document and appears once in the candidate. - Parsed the example argument fragment as JSON after removing its explicit ellipsis placeholder; all four flags exist in the current native argument parser. - Checked the current `o.prompt_cache > 0` live-prefix reuse gate and per-slot cached-state gates in `src/program/generate.cpp`; the prose matches them. - Checked new-section relative links and `git diff --check e88ee39d1d6be6e3364183af3b39b62a7ed6f211...HEAD`; only `docs/BATCHING.md` differs. **Failed:** - None observed in these documentation checks. **Not run:** - A new native handoff generation, model/GPU test or backend build. No runtime source or default changes, and no speedup is claimed. ## Limits This is documentation and source-consistency validation, not measured handoff or hardware-capacity evidence.
Sur le site
Liens install, modèles, releases.