Loss function
In short
A loss function is the scoring rule that tells an AI model how wrong its predictions are, as a single number. Training is the process of adjusting the model to make that number as small as possible.
- Also called
- Loss, Cost function, Objective function, Error function
- Topic
- How AI models work
- Level
- Some background helps
Every machine learning model needs a way to measure its mistakes. Google's machine learning course defines loss as "a numerical metric that describes how wrong a model's predictions are", and says the goal of training is to reduce it to its lowest possible value1.
Different jobs use different loss functions. For predicting numbers, such as house prices, common choices are mean absolute error, the average size of the gaps between predicted and real values, and mean squared error, the average of the squared gaps, which punishes big misses more1. For sorting things into categories, the usual choice is cross-entropy loss, which PyTorch describes as useful when training a classification problem2.
Language models are measured the same way. Kaplan and colleagues' 2020 study of neural language models tracked their performance on cross-entropy loss, and found that this loss falls predictably as model size, training data and compute grow3.
The loss function does not change the model by itself. It produces the number, backpropagation works out how each weight affected it, and gradient descent nudges the weights to lower it4.
A low loss on training data is not the whole story. A model can score very well on the examples it has seen and still do badly on new ones, which is called overfitting. That is why engineers also track loss on separate validation data5.
Example
A model predicts that a flat in Leeds will sell for £210,000 and it sells for £200,000. Under mean absolute error that miss counts as 10,000; under mean squared error it counts as 100,000,000, so big misses dominate the score1.
Why it matters
The loss function decides what a model is actually trying to get right. If it measures the wrong thing, the model will get very good at the wrong thing.
Questions people ask
Is a loss function the same as a cost function?
In everyday use, largely yes. The terms are often used interchangeably, along with objective function. Ruder's widely cited overview of gradient descent, for instance, refers to the quantity being minimised as both the objective function and the cost function4.
What loss function do large language models use?
Cross-entropy loss, the standard loss for choosing between categories, computed from the model's raw output scores (logits) and the correct answer2. Kaplan and colleagues' 2020 scaling laws study measured language model performance on exactly this loss3.
Is a lower loss always better?
On new data, generally yes. On training data, not necessarily. If training loss keeps falling while loss on held-out validation data starts rising, the model is overfitting: memorising its examples instead of learning patterns that carry over5.
Related terms
Sources
- Google for Developers, Machine Learning Crash Course: Linear regression, loss
- PyTorch documentation, torch.nn.CrossEntropyLoss
- Kaplan et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361
- Ruder (2016), An overview of gradient descent optimization algorithms, arXiv:1609.04747
- Google for Developers, Machine Learning Crash Course: Overfitting
- Wald (1939), Contributions to the Theory of Statistical Estimation and Testing Hypotheses, Annals of Mathematical Statistics 10(4), 299-326
- MacTutor History of Mathematics, University of St Andrews: Abraham Wald biography