Memory-Efficient Audio Synthesis Using a Decoupled Temporal-Depth Diffusion Transformer

Siri Expressive Voices, built on Apple's AFM 3 Core Advanced on-device foundation model, generates rich, configurable speech in real time. To achieve this, Apple introduced an efficient detokenizer architecture that operates within the limited compute and memory budget of the AMX (Apple Matrix Coprocessor).

The core technology converts semantic audio tokens into RVQ (Residual Vector Quantization) representations, systematically decoupling temporal and depth processing through a three-component design combining a streaming encoder, a temporal decoder, and a depth decoder. In particular, a single reusable depth decoder using DiT (Diffusion Transformer)-style stage conditioning replaces the conventional multi-decoder structure.

Key technical features include:

• Causal sliding window attention with fixed-window KV caching, maintaining constant memory complexity regardless of sequence length
• Approximately 10ms per generation step on AMX deployment (about 16x faster than real time)
• An extremely small memory footprint of approximately 21MB peak runtime memory and 329MB on-device assets
• Resolves the linear/quadratic memory scaling issues of existing transformer- and GAN-based approaches

This architecture is applied to Siri Expressive Voices on actual Apple devices, allowing users to adjust the Pace and Expressivity of speech. Compared to existing on-device TTS systems, overall MOS (Mean Opinion Score) rose by +0.28 (4.15 vs 3.87), with conversational speech in particular showing a +0.42 (4.24 vs 3.82) performance improvement.