A Walking 3D World

Lyra 2.0 is a framework that generates a walkthrough video following a user-specified camera trajectory, then lifts it into 3D via feed-forward reconstruction.

It directly addresses two common problems in long trajectories and large viewpoint changes: spatial forgetting and temporal drifting.

• For spatial forgetting, it stores per-frame 3D geometry solely for information routing, retrieving relevant past frames to build dense correspondence with the current viewpoint.
• For temporal drifting, it uses self-augmented training that lets the model see its own corrupted outputs, so it learns to correct drift instead of accumulating it.

This approach produces longer, more 3D-consistent video trajectories, and the results are used to fine-tune a high-quality 3D scene reconstruction model.

It also provides an interactive GUI that lets users plan camera paths while viewing the accumulated point cloud, and the generated results can be exported as 3DGS and mesh to connect with physics simulators like NVIDIA Isaac Sim. In other words, the goal is not simple video generation, but generating explorable 3D environments for simulation.