[NeurIPS 2023] Multimodal Generation for Audio and Music

At NeurIPS 2023, research on generating artistic works using AI was actively discussed. In particular, models such as Google DeepMind's Lyria and Meta's Audiobox drew significant attention in the field of audio and music generation.

The Data Intelligence (DI) Lab at LG AI Research also presented a paper titled The Interface for Symbolic Music Loop Generation Conditioned on Musical Metadata. This interface takes metadata such as instrument, tempo, and pitch as input and generates 4-bar MIDI loops, supporting music creators in working more efficiently.

Multimodal Generation generally follows a framework consisting of an Encoder/Decoder structure that compresses data, and a Prior that connects different modalities. In this process, data is compressed via an Autoencoder, and a trained Decoder is used to restore it to its original domain.

To represent data as finite semantic units, Discrete Representation techniques are used. As a representative example, VQ-VAE provides the core principle of representing data using a finite number of Codes during the process of compressing and reconstructing data.