Epoch
In short
An epoch is one pass through the training data. In Keras, that means one iteration over the whole set of examples, unless the trainer is told to stop after a fixed number of batches instead.
- Topic
- How models are trained
- Level
- Beginner
Keras describes fit as training for a fixed number of epochs, and it calls those epochs dataset iterations1. An epoch is an iteration over the entire x and y data provided, unless steps_per_epoch is set to something other than none1. If training resumes with initial_epoch, the epochs argument is the final epoch index. The model is trained until that index, not for that many extra passes counted from zero1.
An epoch is not one weight update. train_on_batch runs a single gradient update on a single batch of data1. batch_size is the number of samples in that update. When it is left unset, and the data are not already delivered in batches, Keras uses 321. For ordinary arrays, the default number of steps in an epoch is the number of samples divided by the batch size, or 1 if that cannot be worked out1. A dataset that repeats forever keeps going unless steps_per_epoch is set1. The history object then records loss and metrics at successive epochs1.
One pass can still be a very short run. Kaplan and colleagues note that a language-model dataset cut to about 2 times 10 to the 7 tokens gives an epoch of only 40 parameter updates2. In their later accounting, training for a single epoch means the run does not reuse examples2.
Large language-model papers do not share one epoch count. Hoffmann and colleagues say the training runs behind their scaling fits all used less than one epoch of data3. In the Chinchilla run of 1.4 trillion tokens, the MassiveWeb and Wikipedia slices were each sampled for more than one epoch3. Nothing in these sources names a universal number of epochs.
Example
Take a table of training rows and a batch size of 32, the Keras default when batch size is left unset1. One epoch is one pass over every row, so a set of 3,200 rows is 100 gradient updates, while train_on_batch would be just one of those updates1.
Why it matters
Training logs report loss per epoch, and people treat a bigger epoch count as more training. The count only means something once you know whether each epoch really saw the whole dataset, and whether examples were repeated.
Questions people ask
Is an epoch the same as a step or a batch?
No. In Keras, train_on_batch is one gradient update on one batch1. An epoch is a pass over the whole dataset, made of many such updates when the batch is smaller than the dataset, unless steps_per_epoch cuts the pass short1.
How many epochs should a model train for?
These sources do not give one number. Kaplan and colleagues describe a single epoch as training without reusing examples2. Hoffmann and colleagues' scaling runs used less than one epoch, while parts of the Chinchilla dataset were seen more than once3.
What happens if the training data never ends?
Keras says that if the dataset repeats forever, you must set steps_per_epoch. Otherwise training runs indefinitely, because an epoch is defined as continuing until the input is exhausted1.
Related terms
Sources
- Keras 3.15.1 model training APIs (fit, train_on_batch, History). GitHub shows the v3.15.1 release as 29 July; the year is not printed on the page
- Kaplan et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361
- Hoffmann et al. (2022), Training Compute-Optimal Large Language Models, arXiv:2203.15556