Pull requests / #1247

#1247 prefill: --prefill-help and --prefill-help-frac as the official flags for the idle-stage helper

open · @anon761 · 0 commentaires · Sur GitHub

Multi-GPUNVIDIA / CUDA

Description

`STRATA_PREFILL_HELP` / `STRATA_PREFILL_HELP_FRAC` already do the work, but env-only is awkward when every other knob has a flag. These two set them before Prefill reads them, so the behavior is identical and repeatable; the envs stay honoured for existing configs.\n\nMeasured on a two-card layer split (2x RTX 3090): about +30% prompt-read time on small prompts (~1.4K tokens), where the second card otherwise sits idle during the first chunk.

Sur le site

Liens install, modèles, releases.