Pull requests / #2421
#2421 Video: the encoder reads the clip itself (ENCV - mtmd samples its frames at video_fps and merges them)
open · @miaomiaozii · 0 comentarios · En GitHub
Descripción
The serve reads video `video_url` parts (OpenAI chat-completions) natively: the vision encoder samples the clip's frames itself (mtmd at `video_fps`, capped by `max_video_frames`) and encodes them merged the model's way, so the clip takes one media place in the prompt - its start marker plus the image tokens its frames expand to. Commits: - cut a clip into its frames (ffmpeg) for the vision path - read the clip in the encoder (ENCV - mtmd samples its frames at video_fps and merges them; one media place per clip) - the encoder reads the clip itself (docstring in the frontend's part list) - test_server: the #2228 Keyed fake keeps the encoder's keep parameter, as the real Vision.encode does Notes: - rebased on main (296b59d2); conflicts in prepare() resolved keeping the upstream `embeddings.keys`/`_vision_up()`/`last_image_at` plumbing around the ENCV loop - full test suite: 1275 pass - e2e on a 6.1s 1920x996 clip: prompt 1349 -> 7813 tokens, reply describes the clip correctly; image path unaffected (1420 tokens)
En el sitio
Enlaces a install, modelos, releases.