VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Existing video tokenization methods represent video as a 3D grid, forcing the model to predict all low-level details at the pixel level regardless of the video's complexity, which leads to high training complexity.

VideoFlexTok represents video as a variable-length token sequence with a Coarse-to-Fine structure. The initial tokens capture abstract information such as semantics and motion, while subsequent tokens add fine-grained details. This allows the generative model to flexibly adjust the number of tokens according to the requirements of downstream tasks.

Key results are as follows:

• Training Efficiency: Achieved similar generation quality (gFVD, ViCLIP Score) with a model 5 times smaller (1.1B vs 5.2B) than the existing 3D grid approach.
• Long Video Generation: When generating a 10-second (81-frame) video, it used only 672 tokens, 8 times fewer than existing methods, significantly reducing computational cost.
• Flexibility: The Generative Flow Decoder enables realistic video reconstruction at any number of tokens.