Runway Unveils Real-Time Generative Interactive World 'GWM Worlds 2'

Runway has unveiled GWM Worlds 2, a real-time generative interactive world simulator. Built on an audio-video generative foundation model, this model generates 720p resolution video at 24fps and 48,000 Hz audio in real time, maintaining spatial consistency even during long movement sequences.

WorldPrompt and Architecture

GWM Worlds 2 uses the WorldPrompt format, which separates the environment into a 'Persistent world context' and a 'Timestamped event stream'. The Genesis prompt defines the environment, physical laws, and object properties, while a timestamped action stream controls movement, dialogue, and camera motion. By post-training a bidirectional model in an autoregressive manner, real-time generation is possible without infinite length limitations, and decoding speed is optimized through a sliding KV cache.

Interaction Methods and Applications

Users can control the world through text actions and continuous camera motions, with immediate responsiveness enabled via key bindings. It supports three usage modes (Ahead of time, Turn-based, Real-time); notably, in Real-time mode, low latency allows keyboard/mouse inputs to be mapped to pre-written prompts and reflected instantly without text input. This can be utilized for interactive entertainment, virtual characters, robotics simulation, and agent training.

Limitations and Outlook

Currently, GWM Worlds 2 is in the research preview stage, sacrificing some fidelity for real-time generation speed. Texture degradation may occur during very fast camera rotations, and long-term memory is incomplete. Runway expects real-time generation technology to mature along a similar curve as offline video generation technology has advanced.