NVIDIA Unveils 4D Interaction Reconstruction Technology from Monocular Video

CARI4D, developed by researchers at NVIDIA and the University of Tübingen, is a technology that reconstructs 4D interactions (temporal changes in 3D shape and pose) between humans and objects at metric scale from a single monocular RGB video.

Unlike previous research that relied on specific object templates or fixed categories, CARI4D takes a category-agnostic approach. This enables zero-shot performance across diverse types of objects.

The core mechanism is as follows:

• Combining Foundation Models: Leverages predictions from existing foundation models in the fields of shape, pose, and scene understanding.
• Spatial Alignment: Aligns predictions from models in different coordinate systems into the same metric-scale space to obtain robust initial values.
• Interaction Optimization: Based on the aligned initial values, infers fine-grained contact and dynamics with the object to precisely refine the pose.

Without requiring any special equipment, this technology can precisely reconstruct complex movements between people and arbitrary objects—such as tires, cement bags, and ski equipment—using only smartphone video, making it highly applicable in robotics and VR/AR fields.