Lift4D: Harmonizing Single-View 3D Estimation for In-the-Wild 4D Reconstruction

Existing 4D reconstruction methods have shown limitations in environments with complex occlusion or large deformations due to the lack of 4D training data or reliance solely on video supervision.

Lift4D introduces the following test-time optimization framework to address this.

• Causal Latent Conditioning: Adapts an existing single-view 3D reconstruction model to generate temporally consistent per-frame predictions, providing a consistent initialization for a 3D Gaussian Splatting representation.
• Occlusion-aware Optimization: 'Sculpts' the representation to match the input video, while completing occluded regions in a visually natural way using a View-conditioned Diffusion Prior.

This methodology has demonstrated superior performance over existing methods, particularly on In-the-wild sequences involving severe occlusion and non-rigid motion.