AI BriefingTRL Adds DPO Support for VLMs
Key point
HuggingFace's TRL library now supports Direct Preference Optimization (DPO) for Vision Language Models (VLMs).
Details
Existing Supervised Fine-Tuning (SFT) approaches require assigning specific labels to data, making them costly, but Preference Optimization can more effectively reflect human judgment in models by comparing rankings between answers.
HuggingFace's TRL (Transformer Reinforcement Learning) library now supports DPO (Direct Preference Optimization) for Vision Language Models (VLMs) as well. This allows developers to train VLMs to more sophisticatedly understand the relationship between visual information and text.
For training, a dataset is needed where 'Chosen' and 'Rejected' answers are paired for questions and images. This guide uses openbmb/RLAIF-V-Dataset to convert data into a conversational template and explains the data formatting process for VLM training.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Apple Releases LensVLM-9B for Processing Compressed Text Images
Meta Releases FLAT, a Variable-Length Continuous Token Model Integrating Image and Text