StepAudio 3 Gen Achieves SOTA in TTS and Multimodal Audio Generation via RVQ-Based Discrete Generation

StepAudio 3 Gen is a general-purpose audio generation model that integrates support for various functions, including zero-shot TTS, voice design, vocals, sound effects, and music. Moving away from the existing continuous generation paradigm based on diffusion Transformers, this model adopts a discrete autoregressive generator architecture that directly models audio on RVQ (residual vector quantization) tokens.

The core component, the StepAudio Tokenizer, represents general audio at a 12.5 Hz frequency and uses a shared residual code space of size 16 x 2048. It is designed to jointly quantize semantic and waveform-level acoustic features, ensuring that each code layer preserves both types of information.

The generation mechanism involves the backbone predicting the first codebook autoregressively along the time axis, while the remaining 15 codebooks are completed along the codebook axis by a lightweight causal Transformer. The model development applied three core design principles: interference-aware progressive pretraining to acquire audio capabilities while preserving the LLM's text capabilities, an RVQ Adaptor for multi-codebook integration, and discrete autoregressive modeling for cross-domain shared representations.

The training process proceeded in the order of interference-aware progressive pretraining, multi-task instruction training, and supervised fine-tuning. As a result, it achieved state-of-the-art performance in TTS and voice design fields, while maintaining strong generation capabilities across speech, vocals, sound effects, and music.