Pre-training

In 2010 Erhan and colleagues wrote that the best supervised results on deep networks then came from an unsupervised pre-training phase¹. The algorithms they discuss, including Hinton and colleagues' 2006 deep belief networks, pre-train one layer at a time so each layer captures the main variation in its input¹. That phase sets the stage for a last phase that fine-tunes the whole network on a labelled objective¹. They argue the first phase works as a regulariser, toward solutions that generalise better¹.

For language, the usual meaning is Radford, Narasimhan, Salimans and Sutskever's 2018 paper². Stage one trains a Transformer decoder as a language model on unlabelled text. The objective is the sum of the log probability of each next token given the tokens before it². They trained on the BooksCorpus, more than 7,000 unpublished books, and called this generative pre-training, followed by discriminative fine-tuning². Devlin and colleagues later gloss that model as the Generative Pre-trained Transformer, OpenAI GPT³.

Pre-training is not one recipe. Radford's model reads left to right and predicts the next token². BERT pre-trains a bidirectional Transformer by masking tokens and predicting them from both sides, plus a next-sentence task. Devlin and colleagues are explicit that this is not a left-to-right language model³. Both train one general model before it is reused. Only the first is next-token prediction²³.

Scale is the expensive part. LLaMA's Pre-training Data section says the models were trained on trillions of tokens of public text⁴. The 6.7 billion and 13 billion parameter models saw 1.0 trillion tokens. The 32.5 billion and 65.2 billion models saw 1.4 trillion⁴. Training the 65 billion parameter model on 2,048 A100 80GB GPUs took about 21 days on that 1.4 trillion token set⁴. Those figures are for LLaMA as published in 2023.

Pre-training does not by itself make a model follow instructions. Ouyang and colleagues write that making language models bigger does not inherently make them better at following intent, so they then fine-tune GPT-3⁵. OpenAI's current docs still separate the steps: models are already pre-trained across a wide range of subjects, and fine-tuning is the later pass on a base model plus the inputs and outputs you want⁶.