Pull requests / #2777

#2777 Start the vision encoder in parallel when it is on another GPU

open · @apatassini · 0 Kommentare · Auf GitHub

NVIDIA / CUDAModels & quants

Beschreibung

`main()` starts and warms up the encoder (~7 s at 4,096 image tokens) before the engine, so that on the same card it takes its
VRAM before the engine sizes its cache. When `vision.cuda_device` is not one of the engine's GPUs that ordering is not needed:
the encoder starts on a thread while the engine loads. Cold start 21 → 14 s together with #3. The HTTP server only serves once
the engine is up, by which time the encoder is ready.

Full notes and measurements: https://github.com/apatassini/Strata/blob/qwen38-16gb/QWEN38.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.