SV4D: Generating Dynamic 3D Content with Multi-Frame and Multi-View Consistency

Existing approaches had to rely on separately trained generative models for video generation and novel view synthesis. To address this, SV4D proposes a unified diffusion model that generates dynamic 3D content from a single video while maintaining multi-frame and multi-view consistency.

When a user inputs a monocular reference video, SV4D generates temporally consistent novel-view videos for each video frame. The generated videos are then used to efficiently optimize an implicit 4D representation such as a dynamic NeRF.

The key point is that this process enables efficient optimization without the cumbersome SDS (Score Distillation Sampling)-based optimization process that prior work mainly relied on. To train the model, a refined dynamic 3D object dataset extracted from the Objaverse dataset was used.

Experiments across various datasets and a user study demonstrated that SV4D achieves SOTA (State-of-the-art) performance that surpasses existing models in novel view video synthesis and 4D generation.