Backpropagation
In short
Backpropagation is the method neural networks use to learn from their mistakes. After the network makes a guess, it works backwards from the error to calculate how much each internal setting (weight) contributed, so every weight can be nudged to do better.
- Also called
- Backprop, Back-propagation, Backward propagation of errors, BP
- Topic
- How AI models work
- Level
- Some background helps
Training a neural network happens in two passes. In the forward pass, data runs through the network's layers and it makes its best guess at the answer. The guess is compared with the right answer to give an error. In the backward pass, the network works back from the output, calculating how the error changes as each parameter changes. Those figures are called gradients3.
Backpropagation is the efficient way to get all those gradients at once. It reuses the calculations from later layers when working out earlier ones, so the cost of the backward pass is about the same as the forward pass. Mathematicians call the same technique the reverse mode of automatic differentiation2.
Backpropagation only works out which way to adjust the weights. The adjusting itself is done by an optimiser such as gradient descent, which moves each weight a small step in the direction that lowers the error3.
The 1986 Nature paper by Rumelhart, Hinton and Williams showed that this procedure lets "hidden" layers inside a network learn useful features of a task on their own, which earlier, simpler methods could not do1. The idea itself is older: a 1970 master's thesis by Seppo Linnainmaa described efficient backpropagation without reference to neural networks, and Paul Werbos applied it to neural networks in 19812.
Modern AI frameworks do it automatically. In PyTorch, a widely used framework for building models, the autograd engine runs backpropagation when a programmer calls .backward() on the error, storing each parameter's gradient ready for the update3.
Example
Imagine a network wrongly labels a photo of a cat as a dog. Backpropagation traces back through the layers and works out, for each of millions of weights, how much nudging it would have made "cat" more likely. Gradient descent then makes those nudges, and the next guess is a little better3.
Why it matters
Without backpropagation, training networks with billions of settings would be impractically slow. It is the engine underneath almost every AI model in use today.
Questions people ask
Who invented backpropagation?
No single person. Seppo Linnainmaa described efficient backpropagation in a 1970 master's thesis, and Paul Werbos first applied it to neural networks in 19812. The 1986 Nature paper by Rumelhart, Hinton and Williams made it widely known, showing that hidden layers could learn useful internal representations12.
Is backpropagation the same as gradient descent?
No. Backpropagation calculates the gradients, meaning how much each weight affected the error. Gradient descent uses those gradients to update the weights. In practice they run together in every training step3.
Why is it called backpropagation?
Because the error is passed backwards. The network compares its output with the right answer, then traverses back from the output layer towards the input, collecting how much each parameter contributed to the error3. The 1986 paper that popularised it was titled "Learning representations by back-propagating errors"1.