3.85x Inference Speedup via Diffusion Draft Generation and AR Verification Based on GLM-OCR

Based on the GLM-OCR (0.9B) model, Self-speculative decoding, a technique where Diffusion generates drafts and the AR (autoregressive) path verifies them, was applied to significantly improve inference speed. This method maintains byte-level accuracy identical to AR output while committing an average of 9.6 tokens per forward pass.

Key Achievements and Performance

• Speed Improvement: Compared to the baseline, decoding-only speed increased by 3.85x, and full-page processing time decreased by 1.34x.
• Accuracy Retention: Achieved a score of 95.16 on OmniDocBench, outperforming the MTP (Multi-token prediction) baseline of the same model.
• Cost Efficiency: While using Diffusion alone introduces a trade-off between accuracy and parallelism, combining it with AR verification yields better results within the same forward budget.

Technical Approach

• OCR-Specific Parallel Generation: Since OCR output text is already determined by the input image, the Diffusion model shows high consistency with the AR decoder (an average of 19 consecutive tokens matching per round) even when predicting multiple tokens simultaneously.
• Block Diffusion Application: To resolve context inconsistency issues arising from processing entire documents at once, Block Diffusion was used, which maintains causal relationships at the block level while filling them in parallel.
• Hybrid Decoding: A single model learns both AR and Block Diffusion views, creating a structure where Diffusion provides fast drafts and AR verifies them to correct errors.