NVIDIA's Lyra 2: Explorable Generative 3D Worlds

Lyra 2.0 works by first creating a walkthrough of a scene using camera-controlled video generation, then lifting it into 3D via feed-forward reconstruction. The goal isn't simply scene generation, but creating explorable 3D worlds capable of real-time rendering and simulation at scale.

Existing video models quickly break down when dealing with long camera trajectories, large viewpoint changes, and revisiting locations already passed. The paper defines this problem in two ways.

• spatial forgetting: when previously seen regions get pushed out of the temporal context, the model ends up hallucinating structure anew upon revisiting
• temporal drifting: as autoregressive generation accumulates small errors, the scene's appearance and geometry gradually distort

To address this, Lyra 2.0 maintains per-frame 3D geometry, but rather than using it directly as a basis for generation, it uses it only for information routing. That is, 3D information is used only to retrieve relevant past frames and build dense correspondences with the target viewpoint, while actual appearance synthesis is left to the generative prior.

It is also trained with self-augmented histories, making the model see its own degraded output. This trains the model to correct drift rather than continuously propagating errors.

Combining these two strategies maintains 3D consistency over much longer trajectories, and the results are used to fine-tune feed-forward reconstruction models, enabling stable, higher-quality 3D scene reconstruction.