Instruction tuning
In short
Instruction tuning is fine-tuning a language model on many tasks written as plain instructions, so it can follow new instructions it was not trained on.
- Also called
- Instruction finetuning
- Topic
- How models are trained
- Level
- Beginner
- First described
- Jason Wei and colleagues at Google, September 2021, in "Finetuned Language Models Are Zero-Shot Learners", which named the method and the FLAN model1.
Pre-training teaches a model to continue text. Instruction tuning is a later step. Wei and colleagues at Google defined it as fine-tuning on a collection of tasks described via instructions, and showed that this improves zero-shot results on tasks the model was not tuned on1. They started from a 137 billion parameter pretrained model, tuned it on more than 60 NLP tasks written as natural language instruction templates, and called the result FLAN1.
On 20 of 25 tasks they measured, FLAN beat zero-shot GPT-3 at 175 billion parameters. It also beat few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA and StoryCloze1. Their ablations say three things matter: how many datasets you fine-tune on, how large the model is, and whether the tasks are written as natural language instructions1.
A follow-up scaled the idea. Chung and colleagues instruction-finetuned PaLM, T5 and U-PaLM, including on chain-of-thought data2. Flan-PaLM 540B, tuned on 1,800 tasks, beat PaLM 540B by 9.4% on average, and scored 75.2% on five-shot MMLU2. They released Flan-T5 checkpoints, which were strong in the few-shot setting even next to larger models such as PaLM 62B2.
It is not the same as typing an instruction when you ask a question. That leaves the model's weights unchanged. Instruction tuning changes the weights by training on many instructed examples1. It is also not reinforcement learning from human feedback. InstructGPT's extra stage uses human rankings, a reward model and the PPO algorithm. The InstructGPT authors compare that pipeline with a separate baseline: GPT-3 fine-tuned on the FLAN instruction datasets3.
The original ChatGPT post describes a first stage of supervised fine-tuning on conversations in which trainers wrote both the user turns and the assistant turns, then a reward model and PPO4. That first stage teaches the model to follow instructions in a dialogue. It is not the multi-task FLAN collection14.
Example
Imagine thousands of training pairs such as 'Translate hello into Spanish' with the answer 'hola', covering translation, questions and short summaries. After instruction tuning, the model can attempt a new instruction, such as 'Explain inflation in two sentences', without extra examples of that task.
Why it matters
It is a large part of why a general chatbot will attempt an instruction it has never seen, instead of only continuing whatever sentence it was given1.
Questions people ask
Is instruction tuning the same as fine-tuning?
It is a kind of fine-tuning. The distinctive part is the data: many tasks written as instructions, so the model handles new instructions without extra examples in the prompt1.
Is instruction tuning the same as RLHF?
No. Instruction tuning is supervised learning on instructed examples1. In InstructGPT, reinforcement learning from human feedback is a later stage that adds human rankings, a reward model and PPO3.
Is instruction tuning the same as prompting?
No. A prompt is text you type when you ask. Instruction tuning changes the model beforehand by training on instructed tasks1.
Related terms
Sources
- Wei et al. (2021), Finetuned Language Models Are Zero-Shot Learners, arXiv:2109.01652
- Chung et al. (2022), Scaling Instruction-Finetuned Language Models, arXiv:2210.11416
- Ouyang et al. (2022), Training language models to follow instructions with human feedback, arXiv:2203.02155
- OpenAI (2022), Introducing ChatGPT