Skip to content

fix(vllm): preserve source video metadata after frame sampling - #10168

Open
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:fix/vllm-video-metadata
Open

taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:fix/vllm-video-metadata

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Preserve source video metadata after load_vllm_video() samples frames, so downstream timestamp consumers describe the frames at their actual times in the source video. This affects the Gemma4 video path through ms-swift's vLLM backend.

The decoder selects source-frame indices with np.linspace(0, total_frames - 1, num_frames, dtype=int), but _video_get_metadata_local() reports consecutive sampled-frame indices and sets fps to duration / num_frames (seconds per frame). Consumers that compute timestamps as frame_idx / fps therefore receive incorrect times, which can become timestamp text in the Gemma4 video prompt.

This three-line fix retains the original frame count and FPS, and reports the same source-frame indices used by the decoder. The existing do_sample_frames flag continues to disable downstream sampling when the loader has already subsampled the video.

Experiment results

Generated a real lossless FFV1 AVI with 300 frames at 30 FPS (10 seconds). Each frame encodes its original index in its pixel colors, allowing metadata indices to be checked against the actual decoded frames.

Sampled frames Last timestamp before Last timestamp after Actual decoded frame time
16 24.0 s 9.9667 s 9.9667 s
32 99.2 s 9.9667 s 9.9667 s

Local regression checks: 30 passed.

  • 28 cases cover requested counts of 1, 16, 32, 300, 400, 0, and -1 through file paths, bytes, base64, and data URIs. Checked decoded frame identities, metadata, timestamps, and sampling flags.
  • 2 cases exercise Gemma4Template.replace_tag, the actual Transformers Gemma4VideoProcessor, and Gemma4Processor.replace_video_token. Both sampled videos preserve their frame indices and end with prompt timestamp 00:09.
  • The regression fails against the original implementation. Changed-file pre-commit and git diff --check pass.

Validation used OpenCV 5.0.0, NumPy 2.5.1, and Transformers 5.14.1. Regression scripts were run locally; this PR contains only the production fix. No full model inference or running vLLM service was used.

Signed-off-by: taking-lying-flat <1615405@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant