WavFlow, Generating Audio Directly in Waveform Space (GitHub Repo)

WavFlow presents a new paradigm that generates synchronized, high-quality audio from video and text inputs directly in the raw waveform space. Unlike existing approaches, it completely bypasses latent compression.

Core technology consists of waveform patchifying and amplitude lifting, which enable stable flow matching on raw audio and use a direct x-prediction approach.

Performance evaluation: On the VGGSound (VT2A) and AudioCaps (T2A) benchmarks, WavFlow performs on par with existing latent-based methods, demonstrating that end-to-end waveform generation is comparable to traditional frameworks in terms of acoustic richness, fidelity, and synchronization.

Usage:

• Installation: set up a conda environment, then run scripts/setup.sh
• Inference: supports CSV format that can take video, text, or both as input
• External weights such as CLIP and Synchformer are automatically downloaded on first run
• Model options: medium/large × 16kHz/44kHz combinations

Checkpoint limitation: Due to organizational policy, production training checkpoints cannot currently be released, and a base checkpoint trained on fully open-source data is in preparation. In the meantime, users can train their own models using the detailed training guide.

License: CC-BY-NC 4.0, built on open-source projects such as MMAudio, JiT, and Synchformer.