ByteDance Open-Sources 'Bernini', an Integrated Framework for Video Generation and Editing

ByteDance has released Bernini, a new framework for video generation and editing. The framework is designed with separated roles for the MLLM planner and the DiT renderer, where the planner predicts target semantic representations in ViT embedding space instead of pixels, and the renderer generates video conditioned on these representations.

Key Technologies and Architecture

• Role Separation: The planner (Qwen2.5-VL-7B-Instruct) decomposes complex instructions and generates semantic embeddings, while the renderer (Wan2.2-T2V-A14B) generates video via Flow Matching in the VAE Latent Space.
• SA-3D RoPE: Adds segment-specific phases to reduce visual input confusion and controls visibility between inputs using a Hybrid Attention Mask.
• Supported Tasks: Supports 6 tasks, including text-to-image (t2i), image-to-image (i2i), text-to-video (t2v), video-to-video (v2v), and reference-based editing (rv2v, r2v).

Performance and Benchmarks

In blind voting results from the self-built Arena platform, Bernini ranked second with a Bradley-Terry score of 1044, trailing the first-place HappyHorse-1.0 (1080) by 36 points. It was evaluated as reaching the top tier of closed-source commercial models, particularly in complex instruction-following capabilities. However, on VBench, the lightweight Bernini-R 1.3B model (84.69) scored higher than the 7+14B series.

Deployment and Usage Environment

• Model Variants: Available in two forms: the full pipeline Bernini and the renderer-only fine-tuned version Bernini-R. Bernini-R is simpler and more stable to install and run.
• Hardware Requirements: Requires a multi-GPU H100 environment and sequence parallelism; consumer single GPUs only support image tasks (--num_frames 1) with limitations.
• License: Both code and model weights are released under the Apache 2.0 license, allowing for commercial use and retraining.