Browse tools

How models are trained

Learning rate

In short

The learning rate is the positive number that sets how far gradient descent moves the weights on each update. Too large a step can raise the training error. Too small a step can leave training slow or stuck.

Also called
step size
Topic
How models are trained
Level
Beginner
First described
Goodfellow, Bengio and Courville define the learning rate as the positive step size in gradient descent, and they trace that method to Cauchy (1847). They do not claim that Cauchy, or their own book, coined the English name1.

Gradient descent proposes a new point by subtracting epsilon times the gradient from the current point1. Goodfellow, Bengio and Courville call epsilon the learning rate, a positive scalar that sets the size of the step1. They trace the method of steepest descent to Cauchy in 1847, and they do not say that Cauchy used this English name1. One popular choice is a small constant1. A line search instead tries several values of epsilon and keeps the one that makes the objective smallest1.

In their practical chapter, the same authors call the learning rate perhaps the most important hyperparameter. If there is time to tune only one, tune the learning rate2. The effective capacity of the model is highest when the rate is right for the problem, not when it is especially large or especially small2. Training error follows a U-shaped curve2. If the rate is too large, gradient descent can raise the training error rather than lower it. In the idealised quadratic case, that happens when the rate is at least twice its best value2. If the rate is too small, training is not only slower but may stay stuck with a high training error. They say that stuck result is poorly understood, and that it would not happen for a convex loss2.

Nothing in these sources is a universal setting. The Keras training docs show an Adam optimiser with learning_rate set to 0.001 only as an example of the compile call, not as a recommended value for every model4. Kaplan and colleagues trained with a linear warmup for 3,000 steps and then a cosine decay down to zero3. They report that results at convergence were largely independent of the learning-rate schedule3.

The learning rate is not the loss, and it is not the batch size. The step is an attempt to decrease the objective that the gradient is taken from1. In Keras, batch size is the number of samples that go into one gradient update, and it defaults to 32 when left unset4.

Example

Goodfellow, Bengio and Courville describe a U-shaped training-error curve: in an idealised quadratic problem, a learning rate at least twice the best value makes gradient descent increase the training error instead of reducing it2. Kaplan and colleagues' language-model runs used a 3,000-step warmup and then a cosine decay to zero, and the final loss depended little on that schedule3.

Why it matters

It is the first number people change when a model will not train. A run that diverges and a run that barely moves can both be the same update rule with a different step size.

Questions people ask

What is a good learning rate?

These sources do not give one value for every model. Goodfellow, Bengio and Courville say a small constant is a popular choice, and that the useful rate is the one that fits the problem, not simply a large or small one12. A Keras snippet sets Adam's learning_rate to 0.001 only as a documentation example4.

What happens if the learning rate is too high?

Gradient descent can increase the training error instead of decreasing it2. In the idealised quadratic case in Goodfellow, Bengio and Courville, that starts when the learning rate is at least twice its best value2.

Does the learning-rate schedule decide the final loss?

Not in the Kaplan et al. language-model runs. They used a 3,000-step linear warmup and then a cosine decay to zero, and they found that results at convergence were largely independent of the schedule3.

Is the learning rate the same as the loss?

No. The learning rate sets the step length in gradient descent1. The loss, or objective, is the quantity that step is trying to reduce1.

Related terms

Gradient descent Backpropagation Loss function Overfitting Fine-tuning Epoch

Sources

  1. Goodfellow, Bengio and Courville (2016), Deep Learning, chapter 4, Numerical Computation, MIT Press
  2. Goodfellow, Bengio and Courville (2016), Deep Learning, chapter 11, Practical Methodology, MIT Press
  3. Kaplan et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361
  4. Keras 3.15.1 model training APIs (compile example and batch_size). GitHub shows the v3.15.1 release as 29 July; the year is not printed on the page

Back to the glossary