Reinforcement learning from human feedback
In short
Reinforcement learning from human feedback (RLHF) is a way of training a chatbot on people's judgements of which reply is better. Those judgements train a reward model, and reinforcement learning then pushes the chatbot toward replies that model scores highly.
- Also called
- RLHF
- Topic
- How models are trained
- Level
- Some background helps
- First described
- Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei introduced learning a reward from human comparisons, then optimising it with reinforcement learning, in a 2017 paper. OpenAI's 2022 InstructGPT paper applied that recipe to language models and used the name reinforcement learning from human feedback21.
RLHF is a training recipe for making a language model follow instructions. OpenAI's InstructGPT paper sets it out in three steps. First, contractors write the replies they want, and the model is fine-tuned on those demonstrations with ordinary supervised learning. Second, contractors rank several replies to the same prompt, and a separate reward model is trained to predict which reply they would prefer. Third, the language model is updated with reinforcement learning, using the PPO algorithm, so that it scores higher on that reward model1.
The comparison idea is older than the name. In 2017 Paul Christiano and colleagues, at OpenAI and DeepMind, trained a reward predictor on human preferences between short clips of behaviour, then let a reinforcement learner optimise that predictor. They did this for Atari games and simulated robot movement, giving feedback on less than 1 percent of the agent's interactions with the environment, and they taught new behaviours with about an hour of a person's time2. InstructGPT describes that work as the origin of the technique, later carried over to summarising text and then to following instructions1.
OpenAI used the same method for the ChatGPT model it introduced on 30 November 2022, with what it called slight differences in how the data were collected. Contractors again ranked alternative replies, and those rankings trained the reward model3. The page now says that post introduced ChatGPT in 2022 and that ChatGPT has changed substantially since3.
It is not supervised fine-tuning on its own. That is only the first step. It is also not the only way to use preference rankings. Direct preference optimisation, introduced by Rafailov and colleagues in 2023, trains on the same kind of human comparisons with a classification loss and does not run a separate reinforcement-learning loop4. A limit remains either way. InstructGPT says the procedure matches the stated preferences of a particular group of contractors and researchers, not a general notion of human values1. OpenAI also wrote that fixing plausible but incorrect answers is hard in part because, during that reinforcement-learning stage, there is no source of truth3.
Example
A contractor sees two replies to 'explain inflation in one paragraph' and picks the clearer one. InstructGPT was trained on a large set of rankings like that: the rankings taught a reward model, and reinforcement learning then steered the chatbot toward replies the reward model preferred1.
Why it matters
InstructGPT used it to move a next-word model toward replies people prefer1. OpenAI then used the same step for the ChatGPT model it released in November 2022, and noted that this stage has no source of truth for whether an answer is correct3.
Questions people ask
Is RLHF the same as reinforcement learning?
It is a form of it. In ordinary reinforcement learning someone writes the reward. In RLHF, people compare model outputs, a reward model learns to imitate those comparisons, and reinforcement learning then trains the language model to raise that score. InstructGPT uses the PPO algorithm for the last step1.
Is RLHF the only way to train a model on human preferences?
No. Direct preference optimisation, from Rafailov and colleagues in 2023, uses the same kind of preference labels but trains with a classification loss. It does not fit a separate reward model and then run a reinforcement-learning loop4.
Was ChatGPT trained with RLHF?
The model OpenAI introduced on 30 November 2022 was. OpenAI said it used reinforcement learning from human feedback, with the same methods as InstructGPT and slight differences in data collection. The same page says ChatGPT has changed substantially since that post3.
Does RLHF stop a chatbot making things up?
It reduces some of that behaviour but does not remove it. On closed-domain tasks in OpenAI's prompt set, where the reply should not add facts absent from the input, InstructGPT did so about half as often as GPT-3, 21 percent against 41 percent, and still made simple mistakes1. OpenAI also wrote that during reinforcement-learning training there is no source of truth3.
Related terms
Tools that use it
Sources
- Ouyang et al. (2022), Training language models to follow instructions with human feedback, arXiv:2203.02155
- Christiano et al. (2017), Deep reinforcement learning from human preferences, arXiv:1706.03741
- OpenAI (30 November 2022), Introducing ChatGPT
- Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv:2305.18290