AI BriefingPreventing OCR Text Degeneration with DPO
Key point
By applying DPO to OCR tasks, text degeneration that is hard to resolve with SFT alone was significantly improved.
Details
The existing SFT (Supervised Fine-Tuning) approach focuses on learning correct answers, but has limits in suppressing Text Degeneration, where the model falls into repetition loops.
According to research by Dharma-AI, adding a DPO (Direct Preference Optimization) stage after SFT reduced degeneration in all tested models, recording an average reduction rate of 59.4% and a maximum of 87.6%.
The reason this achievement is possible lies in DPO's training method.
- Whole-output basis, not token-level: SFT optimizes probabilities token by token, leaving unaddressed the 'Attractor' phenomenon where the model gets stuck on certain tokens, whereas DPO judges the entire output as either 'Correct' or 'Rejected,' directly learning from structural failures.
- DPO as a failure mode mitigation tool: While existing DPO has mainly been used for aligning chatbot responses, this research demonstrates that DPO can be used as a tool to correct specific failure modes of models in objective tasks like OCR.
- Utilizing self-feedback: The model uses failure cases it generated itself (such as repetition loops) as Rejection Pairs to generate the training signal.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Jane Street Research Discovers Non-Monotonic Patterns in Sequence Weight Scaling