Pull requests / #1247
#1247 prefill: --prefill-help and --prefill-help-frac as the official flags for the idle-stage helper
open · @anon761 · 0 Kommentare · Auf GitHub
Beschreibung
`STRATA_PREFILL_HELP` / `STRATA_PREFILL_HELP_FRAC` already do the work, but env-only is awkward when every other knob has a flag. These two set them before Prefill reads them, so the behavior is identical and repeatable; the envs stay honoured for existing configs.\n\nMeasured on a two-card layer split (2x RTX 3090): about +30% prompt-read time on small prompts (~1.4K tokens), where the second card otherwise sits idle during the first chunk.
Mehr auf der Site
Links zu Install, Modellen, Releases.