AI BriefingAccelerating Text-to-Video Generation via Calibrated Sparse Attention
Key point
CalibAtt is a technique that speeds up video generation models by up to 1.58x by identifying unnecessary connections during attention computation.
Details
Recent Diffusion Models have made high-quality video generation possible, but they suffer from slow execution speed due to the computational bottleneck of Spatiotemporal Attention operations in Transformer-based backbones.
The researchers noted that a significant portion of token-to-token connections consistently record negligibly low scores across diverse inputs, and that this pattern repeats. Based on this observation, they propose CalibAtt, a training-free technique.
The key features of CalibAtt are as follows:
- Offline Calibration: Identifies stable block-wise sparsity and repeating patterns regardless of input.
- Optimized Computation: Compiles the identified patterns into attention operations optimized for each layer, head, and diffusion timestep.
- Efficient Inference: During inference, important connections are computed densely, while unnecessary connections are skipped in a hardware-efficient manner.
Experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models showed that CalibAtt achieves up to a 1.58x end-to-end speedup over existing methods, while maintaining video generation quality and text-to-video alignment performance.
