NVIDIA Releases Diffusion LM

NVIDIA has released the Nemotron-Labs Diffusion language model family. Unlike existing autoregressive (AR) methods that generate one token at a time sequentially, the diffusion approach generates multiple tokens in parallel and then iteratively refines them.

It includes text models at 3B, 8B, 14B scales and an 8B VLM, all released under the NVIDIA Nemotron open model license, which permits commercial use. Training code is provided through the Megatron Bridge framework.

It supports three generation modes within a single model:

• Autoregressive mode: sequential left-to-right generation, like existing LLMs
• Diffusion mode: progressive generation in blocks
• Self-speculation mode: drafts candidate tokens via diffusion and verifies them with AR

According to performance measurements, the 8B model showed a 1.2% improvement in average accuracy over Qwen3 8B, and achieved tokens-per-forward-pass (TPF) efficiency 2.6x higher than AR in diffusion mode, and up to 6.4x higher in self-speculation mode.

The training approach is based on Efficient-DLM research, applying both AR and diffusion objective functions simultaneously to a pretrained AR model, adding parallel generation capability while preserving existing abilities. The model was pretrained on 1.3T tokens from the NVIDIA Nemotron dataset and underwent SFT on 45B tokens.

Deployment via SGLang will soon be supported on the main branch, and inference is currently available through a GitHub PR.