HuggingFace Releases Video Dataset Building Toolkit

The performance of video generation models is directly tied to the quality of training data. Like img2dataset, its image dataset building tool, HuggingFace has released a 3-stage pipeline and open-source toolkit for efficiently building video datasets.

Stage 1: Acquisition

• Use yt-dlp to download videos.
• Split long videos into short clips using the Video to Scenes script.

Stage 2: Pre-processing/filtering

• Frame-level: Perform watermark detection (LAION-5B), aesthetic score prediction, and NSFW content detection.
• Video-level: Measure motion score of videos using OpenCV.

Stage 3: Processing

• Use Florence-2 to perform per-frame captioning, object recognition, and OCR tasks.
• Alternatively, models like Qwen2.5-VL can be used to caption the entire video.

As a real-world example, the process of building the crush-smol-v0 dataset demonstrates selecting only high-quality videos by applying strict filtering with watermark probability under 0.1 and aesthetic score of 5.5 or above.