Introspective Diffusion Language Models

Introspective Diffusion Language Models (I-DLM) is an approach that addresses the core weakness of diffusion language models (DLM)—lack of introspective consistency—aiming to achieve quality close to autoregressive (AR) models of the same scale.

The core claim is clear. AR models perform generation and self-verification together within a single forward pass, but existing DLMs, while good at denoising, are weak at self-verification (introspection), which creates a quality gap.

To address this, I-DLM proposes introspective strided decoding (ISD). Within a single forward pass:

• It generates multiple new tokens
• It simultaneously verifies previously generated tokens
• It determines acceptance based on the p/q acceptance criterion.

For training, starting from a pretrained AR model, it is converted into introspective-consistency training using causal attention, logit shift, and all-masked objective. Inference maintains strict causal attention, allowing it to be directly attached to SGLang without requiring separate custom infrastructure.

The results are also presented strongly.

• I-DLM-8B is claimed to be the first DLM to reach quality on par with its AR counterpart of the same scale.
• It reports AIME-24 +26 and LiveCodeBench-v6 +15 compared to LLaDA-2.1-mini(16B).
• It presents a 2.9–4.1x throughput improvement in high-concurrency environments.
• R-ISD, which uses gated LoRA, is described as providing bit-for-bit identical lossless acceleration.

Benchmarks are reported across 15 items, and it is summarized as outperforming previous DLMs across knowledge, math, code, and instruction following overall. In particular, I-DLM-8B presents results surpassing previous DLM figures such as MMLU 82.4, AIME-24 69.6, HumanEval 93.3, and LCB-v6 45.7.

The key point is that it views the bottleneck of existing DLMs not simply as sampling speed, but as a problem of self-consistency + computational efficiency + serving suitability, and presents a structure that improves all of these simultaneously.