Foley Control aligns a frozen latent text-to-audio model to video

Foley Control is a lightweight video-guided Foley approach that freezes pretrained single-modal models as-is and trains only a small cross-attention bridge between the two models.

It connects V-JEPA2's video embeddings to a frozen Stable Audio Open DiT text-to-audio (T2A) model, inserting a compact video cross-attention right after the existing text cross-attention. This way, the prompt determines the overall semantics, while the video more precisely aligns timing and local dynamics.

The key point is that it learns only the audio-video dependency without retraining the audio prior. The frozen backbones each preserve their strong marginals — video itself and text-conditioned audio — while the bridge supplements only the coupling needed for synchronization.

Video tokens pooling is also applied for training stability and memory savings. As a result, with fewer trainable parameters, it shows temporal alignment and semantic alignment competitive with recent multimodal systems, while maintaining prompt-based control and modular extensibility.

• Advantage: Easy to swap or upgrade the encoder or T2A backbone without end-to-end retraining
• Scope: Currently focused on Video-to-Foley, but the same bridge design can be extended to other audio modalities such as speech