Comparing DPO, IPO, and KTO Preference Alignment Methodologies

An analysis compared the performance and characteristics of three major algorithms for aligning LLMs without reinforcement learning (RL): DPO, IPO, and KTO.

Comparison of Key Methodologies:

• DPO (Direct Preference Optimization): Uses a simple loss function that directly optimizes preference data and is easy to implement, but tends to overfit the dataset quickly.
• IPO (Identity Preference Optimisation): Adds a regularization term to DPO to prevent overfitting, allowing the model to converge stably without additional techniques.
• KTO (Kahneman-Tversky Optimisation): Enables training using only individual 'good/bad' labels rather than paired data. This can reduce data construction costs, making it advantageous for practical environments.

Experimental results confirmed that the hyperparameter $eta$, which determines the weight of the reference model, plays a very important role in performance. Additionally, an error in the IPO implementation (the loss summation method) within the TRL library was fixed, achieving performance consistent with the paper.