改动 / #2421

#2421 Video: the encoder reads the clip itself (ENCV - mtmd samples its frames at video_fps and merges them)

open · @miaomiaozii · 0 评论 · 去 GitHub 看

Server & APIModels & quants

说明

The serve reads video `video_url` parts (OpenAI chat-completions) natively: the vision encoder samples the clip's frames itself (mtmd at `video_fps`, capped by `max_video_frames`) and encodes them merged the model's way, so the clip takes one media place in the prompt - its start marker plus the image tokens its frames expand to.

Commits:
- cut a clip into its frames (ffmpeg) for the vision path
- read the clip in the encoder (ENCV - mtmd samples its frames at video_fps and merges them; one media place per clip)
- the encoder reads the clip itself (docstring in the frontend's part list)
- test_server: the #2228 Keyed fake keeps the encoder's keep parameter, as the real Vision.encode does

Notes:
- rebased on main (296b59d2); conflicts in prepare() resolved keeping the upstream `embeddings.keys`/`_vision_up()`/`last_image_at` plumbing around the ENCV loop
- full test suite: 1275 pass
- e2e on a 6.1s 1920x996 clip: prompt 1349 -> 7813 tokens, reply describes the clip correctly; image path unaffected (1420 tokens)

本站相关内容

相关页面的快捷入口。