Overfitting
In short
Overfitting is when an AI model learns its training examples too closely, memorising quirks and noise instead of general patterns. It then performs well on data it has seen but badly on anything new.
- Also called
- Over-fitting, Overfit model
- Topic
- How AI models work
- Level
- Beginner
- First described
- No single inventor. The term was already in use in psychometrics by 1951, when Edward Cureton wrote that least-squares regression "over-fits"5.
The point of training a model is for it to work on data it has never seen. Google's machine learning course describes an overfit model as one that "matches (memorizes) the training set so closely" that it fails to make correct predictions on new data1. The opposite problem, underfitting, is a model that does not even predict its training data well1.
The usual way to spot it is to hold back some data for checking, called a validation set. Early in training, loss on both sets falls together. If training loss keeps falling while validation loss starts to rise, the model is overfitting1.
Google lists two common causes: a training set that does not properly represent real-world data, and a model that is too complex1. Deep networks with very many parameters are especially prone to it2. One well-known 2016 study showed that state-of-the-art convolutional networks could easily fit training data whose labels had been randomly assigned, a pattern with nothing real to learn3.
Engineers use several defences. Dropout, introduced by Srivastava, Hinton and colleagues, randomly switches off units of the network during training so they cannot co-adapt too much; its authors reported that it significantly reduces overfitting2. For language models, Kaplan and colleagues found that how much a model overfits depends in a predictable way on model size and dataset size4.
The idea is far older than modern AI. In the psychometrics literature, a 1951 paper by Edward Cureton warned that least-squares regression "over-fits", fitting the errors in a sample as well as the real trends5.
Example
A pupil who memorises the answers to last year's mock GCSE paper may score full marks on that paper, then struggle in the real exam when the questions change. That pupil has overfitted: the model equivalent is high accuracy on training data and poor accuracy on new data1.
Why it matters
An overfit model can look impressive in testing and then fail in the real world. It is one reason to be sceptical when an AI system is only reported to do well on the data its makers used to build it.
Questions people ask
What is the difference between overfitting and underfitting?
An overfit model does well on its training data but badly on new data, because it has memorised rather than generalised. An underfit model does badly even on its training data, because it has not learned enough1.
How do you know if a model is overfitting?
Compare its loss on the training data with its loss on a separate validation set. If the two curves fall together and then split, with training loss still dropping but validation loss rising, the model is overfitting1.
How do you prevent overfitting?
Common approaches include training on more representative data, using a simpler model, and regularisation techniques such as dropout, which randomly drops units during training. Its authors reported that dropout significantly reduces overfitting compared with other regularisation methods12.
Related terms
Sources
- Google for Developers, Machine Learning Crash Course: Overfitting
- Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 15(56), 1929-1958
- Zhang, Bengio, Hardt, Recht et al. (2016), Understanding deep learning requires rethinking generalization, arXiv:1611.03530
- Kaplan et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361
- de Rooij et al. (2019), The Early Roots of Statistical Learning in the Psychometric Literature, arXiv:1911.11463 (quoting Cureton 1951, Educational and Psychological Measurement 11:12-15)