NVIDIA and KAIST Introduce BirdDreams, a Driving World Model That Jointly Predicts RGB and BEV Video

·2026.11.16 09:00

Key point

The model uses joint attention between perspective and bird's-eye-view streams to generate realistic traffic reactions and follow ego trajectories with 35% lower error than baselines.

1 / 9

Details

  • BirdDreams is an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video from a single camera image, without requiring HD maps or surrounding-vehicle state inputs at inference.
  • The model achieves 35% lower error (3.83 m vs 5.88 m ADE) in following input ego trajectories compared to an OmniDreams-based baseline on the PhysicalAI-AV dataset.
  • Without ego-action inputs, BirdDreams generates trajectories with 21% lower error (6.15 m vs 7.76 m minADE) than the baseline, producing motion closer to recorded driving.
  • Human evaluation with 25 participants and 375 pairwise judgments found 57% of viewers preferred BirdDreams for realistic surrounding-vehicle behavior over the baseline.

Architecture and Control

  • Two causal video diffusion transformers operate in parallel, one for perspective video and one for BEV, exchanging information via joint attention.
  • Ego control is achieved by translating and rotating the latest BEV frame along an input trajectory to create a soft condition for generation.
  • The BEV stream depicts roads and vehicles as simple shapes at a fixed metric scale, providing explicit geometric structure that the perspective stream uses for visual cues.

Performance and Evaluation

  • Evaluated on PhysicalAI-AV and nuScenes datasets.
  • Best-of-six minADE at 3 seconds is 0.295 m for BirdDreams compared to 0.318 m for the OmniDreams-based baseline.
  • The model can predict reactive traffic behavior, such as vehicles braking or changing lanes to avoid the ego vehicle, without relying on log replays.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.