LAFUFU: Latent Acoustic Features for Ultra-Fast Speech Restoration

Existing Speech Restoration technology aims to reconstruct audio containing noise or reverberation into a clean state. Recently, Generative Diffusion Models have shown excellent performance, but they have the drawback of very high computational cost due to high-dimensional audio representations and iterative sampling processes. In particular, for 48kHz high-resolution audio, the computational burden becomes even greater, limiting applicability to real-time services.

LAFUFU introduces a Latent Space-based acoustic representation method to solve this problem. Instead of performing the diffusion process directly on the raw audio spectrogram, it carries out the diffusion process on Compact Features extracted through a custom-built Autoencoder.

This approach provides the following benefits:

• Improved Inference Speed: By performing computations in a compressed latent space, it achieves very fast speeds even when processing high-resolution audio.
• High-Quality Output: Under the same time constraints, it generates higher quality speech than existing non-latent methods.
• Excellent Benchmark Performance: It has demonstrated competitive performance on the latest benchmarks such as EARS-WHAM and EARS-Reverb 48kHz.