Fine-tuning

In 2010 Erhan and colleagues describe deep belief networks and stacked autoencoders as greedy layer-wise unsupervised pre-training followed by supervised fine-tuning¹. Fine-tuning is the last phase, when gradient updates train the network on a labelled objective¹.

For text, Howard and Ruder set out universal language model fine-tuning in 2018². A language model is trained on a broad corpus, then fine-tuned on the target task². With only 100 labelled examples, they report, it matched training from scratch on 100 times more data². The failure they report is overfitting on small datasets and catastrophic forgetting when a classifier is added². Their remedies were a different learning rate per layer, a rate that rises then falls, and unfreezing layers one at a time from the top².

Radford and colleagues use the same split with a Transformer³. After generative pre-training, they adapt the parameters on a labelled set, add a linear output layer, and maximise the probability of the correct label, while keeping some of the language-model loss³. BERT fine-tunes every pre-trained parameter, plus one new output layer, separately for each downstream task⁴.

Full fine-tuning updates the weights. Hu and colleagues define the ordinary method that way: adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model⁵. LoRA freezes those weights and trains small added matrices. On GPT-3 with 175 billion parameters, they report 10,000 times fewer trainable parameters and one third of the GPU memory of full fine-tuning with Adam⁵. Prompting is separate. OpenAI lists prompt engineering and fine-tuning as different steps, and only fine-tuning trains on a dataset of examples⁶.

Ouyang and colleagues fine-tuned GPT-3 on labeler demonstrations with supervised learning, then again with reinforcement learning from human feedback, and called the result InstructGPT⁷. OpenAI's docs say the company is winding the fine-tuning platform down. New users cannot access it. Existing users can still create training jobs for the coming months, and a fine-tuned model stays available for inference until its base model is deprecated⁶. The same page still defines fine-tuning as taking a pre-trained base model and supplying the inputs and outputs you expect, including more examples than fit in one request⁶.