Gradient descent

In short

Gradient descent is the step-by-step method AI models use to learn. It measures how wrong the model is, works out which direction of change reduces that error, takes a small step that way, and repeats until the error stops falling.

Also called: Stochastic gradient descent, SGD, Steepest descent
Topic: How AI models work
Level: Some background helps
First described: Augustin-Louis Cauchy proposed the gradient method in a note to the French Academy of Sciences dated 18 October 1847, to solve equations from astronomy³.

Training a model means finding settings (weights) that make its error, called the loss, as small as possible. Gradient descent does this by trial and adjustment. Google's machine learning course sets out the loop: calculate the loss, work out which direction to move the weights to reduce it, move them a small amount, and repeat until the loss stops going down¹. At that point the model is said to have converged¹.

The "gradient" is the slope of the loss with respect to each weight. The method updates the weights in the opposite direction to the gradient, like walking downhill until you reach a valley. The size of each step is set by the learning rate².

There are three main versions. Batch gradient descent uses the whole training set for every step. Stochastic gradient descent (SGD) updates after each single example, which is faster but noisier. Mini-batch gradient descent, which updates on small groups of examples, is the usual choice for neural networks, and people often call it SGD too².

It is not guaranteed to find the best possible answer. On simple, bowl-shaped problems it reaches the lowest point, but on the bumpy loss surfaces of neural networks it may settle in a local low point instead². Optimisers such as Adam are refinements of the same idea that adjust the step size for each weight²⁴.

In neural networks, the gradients come from backpropagation, and gradient descent then uses them to update the weights⁵.

Example

Picture walking down a hill in thick fog on the Brecon Beacons. You cannot see the bottom, so you feel which way the ground slopes under your feet and take a step downhill, then check again. Gradient descent does the same thing with a model's error².

Why it matters

It is how neural networks, including the large language models behind chatbots, are trained. When people say a model "learned" something, gradient descent did the learning.

Questions people ask

What is the difference between gradient descent and stochastic gradient descent?

Plain (batch) gradient descent looks at the whole training set before each update. Stochastic gradient descent updates after every single example, which is quicker but noisier. Most neural networks use mini-batches, a middle path, and the name SGD usually covers that too².

What is the learning rate in gradient descent?

The learning rate sets how big a step the method takes downhill each time². Too small and training is painfully slow; too large and the loss can bounce around the minimum or even get worse².

Who invented gradient descent?

The French mathematician Augustin-Louis Cauchy, in a short paper to the Academy of Sciences in October 1847. He wanted a general way to solve the systems of equations that came up when calculating the orbits of heavenly bodies³.

Related terms

Sources

Back to the glossary