Direct Preference Optimization (DPO)

Understanding Its Function

DPO directly optimizes the model using data that indicates "which answer is preferred," and the standard formulation does not require training a separate reward model before running PPO.

Understanding with an Example

For the same question, there are two candidate answers, and the teacher selects the one more suitable for the grade level. DPO learns from such pairwise preferences, making the model more inclined toward the preferred response relative to the reference model.

Going Deeper

Training typically uses questions, preferred answers, and less-preferred answers, along with a fixed reference policy. The goal is to compare the relative probabilities of answers, rather than simply copying the preferred answer as the sole correct answer. Internally, gradients are still computed and the optimizer updates the model.

What It Does Not Tell Us

Not using a separate reward model does not mean high-quality preference data is unnecessary or that there are no implicit reward relationships. Preferences may contain noise, style bias, or factual errors. DPO and PPO are alternative approaches, not a mandatory sequential process.

Next Steps

• Reinforcement Learning from Human Feedback (RLHF)
• Supervised Fine-Tuning (SFT)
• Optimizer

References

• Direct Preference Optimization: A direct objective on preference pairs; the standard formulation requires neither a separately trained reward model nor an online PPO loop.

Information verified on 2026-09-09; the original paper is used to explain the mechanism, and the examples in the text are for instructional purposes.