Diffusion Language Model (dLLM) Research: Overcoming AR Model Speed Limits with Parallel Generation and Commercialization Trends like Mercury 2

Research led by Volodymyr Kuleshov and others covers the technological evolution and commercialization status of Diffusion LLMs (dLLMs) aimed at overcoming the sequential generation limitations of existing autoregressive (AR) LLMs. dLLMs enable error correction and speed improvements by iteratively refining entire sequences in parallel.

Key Technologies and Mechanisms

• Masked Diffusion: Since Gaussian noise cannot be applied to discrete tokens, this approach uses a bidirectional transformer similar to BERT to restore randomly masked tokens.
• Structural Expansion: To overcome the fixed-length and speed limitations of standard MDLMs, Block Diffusion (Gemma Diffusion, generating 256-token blocks), Set Diffusion, and encoder-decoder structures have been introduced.
• Error Correction: Through the Remasking technique, the MAUVE score rises from 0.40 to 0.66 as sampling steps increase, approaching the AR model score (0.76).
• Uniform State Diffusion (UDLM): Using random token replacement instead of masking allows for the modification of all tokens, a method adopted by Gemma Diffusion.

Performance and Commercialization Cases

• Mercury 2 (Inception): Achieved 1,200 tok/sec solely through algorithmic optimization without dedicated chips, delivering quality equivalent to Claude Haiku and Gemini Flash-Lite while being 5–10 times faster.
• Nemotron Diffusion (NVIDIA): A 35B parameter AR-diffusion hybrid model that records 2–8x throughput compared to AR models while maintaining 99% of the quality.
• Gemma Diffusion (Google): Adopted a UDLM backbone and block diffusion to shift memory bandwidth bottlenecks into compute bottlenecks.
• LLaDA: An 8B parameter open-weight model that demonstrated competitiveness against LLaMA2/3 in general, math, and code benchmarks.

Applications and Outlook

• Biology: ESM3 (100B parameters) achieved SOTA in protein generation, and NT-v3 showed superior enhancer expression regulation capabilities compared to native sequences in wet-lab validation.
• Inference Acceleration: Demonstrated up to 10x speed improvement over AR models through parallel processing, evaluated as playing a role similar to how Transformers overcame the limitations of RNNs.