Direct preference optimisation
In short
Direct preference optimisation, or DPO, trains a chatbot on pairs of replies where a person picked the better one, without a separate reward model or a reinforcement learning loop.
- Also called
- DPO, Direct Preference Optimization
- Topic
- How models are trained
- Level
- Some background helps
- First described
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn, May 2023, in "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"1.
The usual preference recipe first trains a reward model on human rankings, then uses reinforcement learning to raise that score without drifting too far from the starting model. Rafailov and colleagues showed that this same objective has a closed-form best policy, so the chatbot can be trained with a simple classification loss instead1. The loss is binary cross-entropy. It raises the probability of the preferred reply and lowers the probability of the rejected one1.
There is still a notion of reward, but it is not a separate network. Under the Bradley-Terry model of preferences, the reward is read off the chatbot: how much more likely a reply is under the new model than under the reference model it started from1. The paper's title states the point directly. The language model is secretly a reward model1. During this fine-tune the method does not sample new replies from the language model, and it does not run reinforcement learning1.
On the authors' tests, DPO beat PPO-based reinforcement learning from human feedback at controlling the sentiment of generated text, and it matched or improved quality on summarisation and single-turn dialogue. The language models in those experiments had up to 6 billion parameters1.
Labs did not all stop training reward models. Meta's Llama 3 paper says post-training uses supervised fine-tuning, rejection sampling and DPO, rather than more complex reinforcement learning. They still train a reward model on human rankings and use it for rejection sampling, then run DPO. They tried PPO and report that DPO used less compute and did better on instruction-following benchmarks2.
DPO is not instruction tuning. The DPO paper treats instruction tuning, fine-tuning on demonstrations of a desired answer, as an earlier stage. Learning from preferences, whether by reinforcement learning or by DPO, comes after that1.
Example
A cooking site shows two answers to 'how do I brown onions without burning them?' A reader marks the clearer one. DPO raises the chance the model writes in that style. Nobody trains a second model just to score the two answers.
Why it matters
It is a practical way to do the last stage of chatbot training when a full reinforcement learning run is hard to scale. Meta reported that, for Llama 3, it needed less compute than PPO2.
Questions people ask
Is DPO the same as RLHF?
It targets the same kind of objective, a higher reward without drifting too far from a reference model, but it skips the reinforcement learning loop. The reward is implicit in the chatbot's probabilities1.
Does DPO need a reward model?
Not a separate one. The method avoids fitting a standalone reward model, and reads an implicit reward from the chatbot itself1. A lab can still train a reward model for other steps. Llama 3 does, and uses it for rejection sampling before DPO2.
Is DPO the same as instruction tuning?
No. Instruction tuning fine-tunes on demonstrations of a desired answer. DPO fine-tunes on a preference between two answers. The DPO paper treats instruction tuning as the earlier stage1.