AI BriefingOrthrus-Qwen3: Up to 7.8x tokens/forward on Qwen3, identical output distribution
Key point
Orthrus, based on Qwen3, achieved up to 7.8x speedup through lossless parallel generation.
Details
Orthrus is a framework that adds dual-view diffusion on top of the Qwen3 backbone, combining the accuracy of autoregressive LLMs with the parallel generation of diffusion models. It touts strictly lossless generation, explaining that it maintains output identical to the base model's prediction distribution.
- Released models: Qwen3-1.7B / 4B / 8B
- Average speedup: 4.25x, 5.20x, 5.36x
- Generation efficiency: presents up to 7.8x increase in verified tokens per forward
- Training method: freezes the base LLM and fine-tunes only 16% of total parameters
- Architecture: the autoregressive view and diffusion view share the same KV cache, keeping memory overhead at O(1) level
In comparative experiments, it emphasizes the balance of speed and accuracy on long context and MATH-500 compared to speculative decoding approaches like EAGLE-3, DFlash and other diffusion LLMs. Native integration with vLLM and SGLang is planned.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
3.85x Inference Speedup via Diffusion Draft Generation and AR Verification Based on GLM-OCRUnderdog Releases 'Husky', a Model-Specific Inference Engine Up to 4.5x Faster than MLX