Seoul National University Researchers Release Audio-Only LM

Researchers at Seoul National University have released Sori-1B, a 1B-parameter audio-language model trained from scratch on a decoder using only paired audio-text data, without text-only pre-training or initialization from existing LMs. This model is designed to address the issue where text-based pre-trained models maintain performance due to text bias even in the absence of audio, and to generate responses grounded in actual audio.

Model Architecture and Training Method

• Encoder: Reuses NVIDIA's Audio Flamingo Next encoder in a frozen state, which accounts for 61.5% of the total parameters.
• Decoder and Tokenizer: The decoder, embeddings, output head, and a custom 'auditory-ontology' tokenizer built on audio concept categories were all trained from scratch.
• Training Data and Resources: Used approximately 7.4k hours and 4.75 million examples, trained on 3 RTX 4090s.

Performance and License

Sori-1B supports multiple-choice question (MCQ), open QA, captioning, and speech recognition (ASR) modes. The research team pointed out that the existing AF3 model retains 74% of its MMAU score even when audio is replaced with silence, suggesting that Sori-1B learns knowledge more faithful to audio. However, due to the reuse of the NVIDIA encoder, a non-commercial/academic-only license applies, and the repository is currently marked as 'coming soon' with weights released on a limited basis.