Self-supervised learning

In short

Self-supervised learning is a way of training a model by hiding part of the input and asking the model to fill that part in. The correct answer is taken from the data itself, so people do not have to label every example.

Also called: SSL, Self-supervision
Topic: How models are trained
Level: Some background helps
First described: A paper that coined the name was not identified. LeCun and Misra's March 2021 essay states the definition now widely cited: predict a hidden part of the input from a visible part. The masked-word version was already in Google's BERT paper in 2018, which called the task unsupervised²³.

Self-supervised learning trains a model on a task built out of the data, rather than on labels a person has written. NIST defines it as learning from implicit labels generated from unstructured data, not from explicit human-made labels. A common task, NIST says, is to show the model only part of an example and ask it to produce the rest¹. Yann LeCun and Ishan Misra put the same idea this way: predict any hidden part of the input from any visible part².

In language, the usual version hides words. Google's BERT paper masks about 15 percent of the word-pieces in a passage at random and trains the model to predict the original words from the surrounding text. The paper called this, and a second task about whether one sentence follows another, unsupervised pre-training on unlabelled text³. LeCun and Misra later described that masked-word training as self-supervised. They argue that 'unsupervised' suggests there is no teaching signal, whereas filling in a blank supplies a great many signals².

NIST uses the term for the pre-training of foundation models, including the GPT family, which it says are pre-trained this way on large sets of unlabelled text¹. OpenAI's November 2022 ChatGPT was fine-tuned from a model in the GPT-3.5 series⁴. Fine-tuning for a specific job can still use ordinary labelled data afterwards. BERT's authors describe that second step as using labelled data from the downstream task³. NIST likewise describes fine-tuning as a later phase, often itself supervised¹.

It is not clustering. Grouping similar items without predicting a hidden piece is what NIST calls unsupervised learning¹. And it is not a person marking up photos or sentences one by one. That is supervised learning, which needs an explicit label for each example¹.

Example

Take the sentence 'The lion chases the blank in the savanna.' A self-supervised model is trained to fill the blank from the words around it. LeCun and Misra use this kind of gap-filling as the basic picture of the method².

Why it matters

NIST describes it as the pre-training method for the GPT family, the predominant design of large language models, which learn from unlabelled text before a later fine-tuning step¹. Filling a blank is not the same as checking that a sentence is true.

Questions people ask

Is self-supervised learning the same as unsupervised learning?

NIST defines them separately. Unsupervised learning finds patterns such as clusters in unlabelled data. Self-supervised learning builds a prediction task, such as filling in a missing part, so the data supplies its own target¹. BERT's authors called their masked-word task unsupervised³. LeCun and Misra say that name is misleading².

How does self-supervised learning work in a chatbot?

NIST says the GPT family, the design behind many chatbots, is pre-trained by self-supervised learning on unlabelled text¹. One version, used by BERT, hides about 15 percent of the word-pieces and trains the model to guess them from context³. A later fine-tuning step can still use human-written labels¹³.

Does self-supervised learning need labelled data?

Not for the pre-training task. NIST says the labels are implicit, inferred from the data, rather than written by people¹. A later fine-tuning stage, for a job such as question answering, often does use labelled examples¹³.

Related terms

Tools that use it

ChatGPTOpenAI's general chatbot, the most used AI app in the UK and worldwide.

Sources

Back to the glossary