Emergent abilities

In short

Emergent abilities are skills a language model shows at a large size but not at a smaller size, so you cannot forecast them just by extending the smaller models' scores.

Also called: Emergence, Emergent capabilities
Topic: How models are trained
Level: Some background helps
First described: Wei and colleagues, in "Emergent Abilities of Large Language Models" (2022). They adapt a broader idea of emergence from Philip Anderson's 1972 essay "More Is Different", and define the language-model version themselves¹.

In 2022 Wei and colleagues defined an ability as emergent when smaller language models lack it and larger ones have it, so it cannot be forecast by extending the smaller models' scores. Plotted against scale, the score sits near random and then rises sharply. That is a different pattern from training loss, which scaling-law papers had found can be predicted as compute grows¹.

One case they plot is few-shot arithmetic: three-digit addition and subtraction, and two-digit multiplication, with a couple of worked examples in the prompt and no further training. GPT-3 and LaMDA stayed near zero across a wide span of training compute. GPT-3's score then jumped at about 2×10^22 FLOPs, a count of training compute, which was the 13-billion-parameter model. On LaMDA, asking the model to write out steps (chain-of-thought prompting) only beat a direct answer on maths word problems once training reached about 10^23 FLOPs, roughly 100 billion parameters¹.

The same paper says the measuring stick can create the cliff. If a long answer scores zero unless every piece is right, steady progress looks like a sudden success. On six tasks they checked, the loss on the correct answer was already falling while the published score was still near chance. They call that an incomplete account, because some classification tests still jumped¹.

Schaeffer, Miranda and Koyejo argued in 2023 that the cliff is often the metric. More than 92% of BIG-Bench tasks marked emergent used exact string match or a right-or-wrong multiple-choice grade². BIG-Bench is a crowd-sourced suite of more than 200 language tests¹. On multiplication and addition in the GPT-3 family, accuracy looked abrupt, but a score that credits partly correct answers rose smoothly. With more test questions, even accuracy was above chance and smooth².

The scale of a jump is also not fixed. Wei and colleagues report 14 BIG-Bench tasks where LaMDA at 137 billion parameters and GPT-3 at 175 billion stayed near chance, while PaLM at 62 billion scored above chance. They point to training data and model design, not parameter count alone¹.

Example

On a few-shot test of three-digit addition and subtraction and two-digit multiplication, GPT-3 stayed near zero until the 13-billion-parameter model, at about 2×10^22 FLOPs of training compute, where the score jumped well above chance¹.

Why it matters

A small model failing a task does not tell you a larger one will fail it. A sudden jump on a pass-or-fail score is also not, by itself, proof that a new kind of skill switched on.

Questions people ask

Are emergent abilities real or a trick of the score?

Both papers agree that some published curves jump. Wei and colleagues treat that jump as a real pattern, while noting that all-or-nothing scores hide partial progress¹. Schaeffer, Miranda and Koyejo show that switching to a smoother score, or using more test questions, can make the same outputs look like a steady climb².

Is an emergent ability the same as grokking?

No. Emergent abilities, as Wei and colleagues define them, are about a skill that appears once models are large enough¹. Grokking, named by Power and colleagues, is one small network that memorises a symbolic dataset and only later answers new questions³.

Can you predict which abilities will appear?

Not from a straight line through small models' scores. That unpredictability is part of Wei and colleagues' definition¹. The scale is not a law of nature either: they report tasks where a 62-billion-parameter PaLM model beat much larger earlier models, which they tie to data and design¹.

Related terms

Sources

Back to the glossary