Orthrus: Diffusion-based Parallel Token Generation Technique Unveiled

Orthrus is a technology that performs parallel token generation by injecting a trainable Diffusion Attention module into frozen AR Transformer layers.

Core Mechanism:

• Dual-View Structure: The Diffusion head and AR head share a single KV cache.
• Parallel Generation and Verification: When the Diffusion head projects 32 tokens in parallel, the AR head verifies them in a second pass and adopts the longest matching prefix.
• Accuracy Preservation: Since the output distribution is mathematically identical to that of the base model, there is no loss of model accuracy, unlike existing Diffusion LMs.

Key Results:

• Speed Improvement: On the MATH-500 benchmark, it improved token generation speed (TPF) by up to 7.8x and wall-clock time by about 6x.
• Efficiency: Unlike Speculative Decoding, it requires no separate Drafter model or additional cache, resulting in no TTFT (Time-To-First-Token) penalty, with very low KV overhead of about 4.5 MiB.
• Training Efficiency: Only 16% of total parameters are trained, and training can be completed within 24 hours on an 8×H200 setup.

Limitations:

• It inherits the biases, hallucinations, and knowledge gaps of the base model as-is.
• It has currently only been evaluated on Qwen3 models, and only supports Greedy and Rejection Sampling methods.