Video Generation Models as General-Purpose Vision Models

While conventional computer vision has relied on task-specific specialized models, GenCeption aims for a paradigm shift in visual intelligence, similar to how NLP evolved into general-purpose language models.

GenCeption is built on a pretrained Video Generative Diffusion Model, leveraging its rich spatiotemporal world knowledge (World Priors) and vision-language alignment capabilities. It is then converted into a feed-forward model capable of performing various cognition tasks through synthetic-data-driven multitask Post-training.

Key achievements include the following:

• Achieving SOTA performance: It shows performance comparable to or better than existing specialist models across various tasks such as Depth, Surface Normal, Camera Pose, and Segmentation.
• Overwhelming data efficiency: It reaches similar accuracy using only 7x to up to 500x fewer training frames compared to existing models.
• Generalization capability: It demonstrates excellent generalization performance in transferring knowledge learned from simulation to real environments (Sim-to-real transfer) and to novel object categories not present in the training data.