Video Generation Models as World Simulators

OpenAI trains large-scale generative models using video and image data with various resolutions and aspect ratios. Sora, the core of this technology, can generate high-quality videos up to a minute long, and aims to build a general-purpose simulator of the physical world by scaling video generation models.

To process visual data, it uses the visual patches approach, borrowing the token concept from LLMs. A video is first compressed into a lower-dimensional latent space, then decomposed into spatiotemporal patches (spacetime patches) that can be learned by a transformer.

Sora adopts a diffusion transformer architecture. By processing video data in units of spacetime patches, this provides the following benefits.

• Support for various formats: It can flexibly train on and generate videos with various resolutions, durations, and aspect ratios.
• Improved composition and framing: By training on data in its original aspect ratio, it reduces subject cropping and forms natural compositions.
• Sampling flexibility: It can directly generate content optimized for various devices, from widescreen to vertical video.