Beyond Next-Token Prediction: Performance Characteristics of Diffusion and Autoregressive Language Models

Autoregressive language models (ARMs) generate tokens one by one based on previous tokens, resulting in high sequential dependency and low arithmetic intensity. In contrast, Diffusion language models (DLMs) offer the potential for faster inference by generating multiple tokens in parallel.

Analysis revealed that DLMs can achieve higher arithmetic intensity than ARMs by leveraging parallelism across token positions, but they failed to scale efficiently as context length increased. To address this, researchers examined block-wise decoding, which decouples sequence length from arithmetic intensity, and confirmed scalability similar to ARMs in long contexts.

In batch inference, ARMs demonstrated higher throughput by more effectively utilizing parallelism across multiple sequences. Additionally, reducing the number of sampling steps while maintaining generation quality was identified as a key challenge for open-source DLMs to achieve lower latency than ARMs.