DLoop: Looped Speculative Decoding Improves LLM Inference Speed by 5-41%

DLoop introduces a looped form of speculative decoding that adaptively performs multiple drafting stages before verification, reducing the number of expensive target-model forward passes. As draft models become more capable, they often produce tokens that are fully accepted by the target model, making the subsequent verification step redundant if drafting could have continued. DLoop addresses this by continuing to draft while the model remains confident and verifying all accumulated tokens together.

How It Works

• Adaptive Drafting: Unlike standard methods that verify after each stage, DLoop loops through multiple drafting stages based on confidence.
• Loop-Aware Training: The draft model is trained to remain reliable during these additional stages by being exposed to its own hidden states for unverified draft tokens.
• Efficiency: The method trades additional draft-model forward passes for a reduction in target-model forward passes.

Performance Impact

DLoop integrates with diverse speculative decoding methods, including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules. It improves wall-clock speedup by 5 to 41 percent while preserving lossless decoding. The code is currently under internal review and will be released soon via GitHub.