FLOPs

In short

FLOPs are a count of floating-point operations, such as the additions and multiplications used to train a model. The count is not a chip's peak speed. A peak rate is usually written FLOPS, meaning operations per second.

Also called: FLOP, floating-point operations
Topic: How models are trained
Level: Some background helps
First described: No first use of the abbreviation was identified here. OpenAI's archived analysis already counted training FLOPs as adds and multiplies, with two FLOPs per add-multiply and a factor of three for the backward pass⁵. Kaplan and colleagues later wrote the transformer form of about 6N operations per token².

A FLOP is one floating-point operation⁵. FLOPs, with a lowercase s, is how many of those operations a training run used⁵. When operations could not be counted directly, OpenAI estimated them from GPU time, and the authors say this is not a count of peak theoretical FLOPS⁵. The estimate uses an assumed share of that peak, often one third for a GPU, to guess how many operations ran⁵.

When the architecture is known, the same note counts adds and multiplies in one forward pass, multiplies by 2 because each add-multiply is two FLOPs, by 3 for the forward and backward passes, and by the number of examples and epochs⁵. The aim was to be right within a factor of about 2 to 3⁵. For AlexNet, counting operations gave 0.0054 petaflop/s-days and the GPU-time estimate gave 0.0058⁵.

Kaplan and colleagues use the same factor of two for transformers². A forward pass is about 2N operations, where N leaves out embedding parameters, because each multiply-accumulate counts as two operations². The backward pass is about twice the forward pass, so training is about 6N floating-point operations per token². One petaflop-day is 8.64 times 10 to the 19 operations². Hoffmann and colleagues use the matching total, about 6ND for a model of size N trained on D tokens³.

In the Chinchilla paper, Gopher's training compute is 5.76 times 10 to the 23 FLOPs, and Chinchilla used that same number of operations with 70 billion parameters and 1.4 trillion tokens, against Gopher's 280 billion parameters and 300 billion tokens³. Sevilla and colleagues' Table 2 gives three ranges through 2022: about 3 times 10 to the 4 up to 2 times 10 to the 14 FLOPs before 2010, about 7 times 10 to the 14 up to 2 times 10 to the 18 in the deep-learning era, and about 4 times 10 to the 21 up to 8 times 10 to the 23 for the large-scale group¹.

Article 51 of the EU AI Act presumes a general-purpose model has high-impact capabilities when cumulative training computation, measured in floating-point operations, is greater than 10 to the 25⁴.

Example

OpenAI's recount of AlexNet shows the gap between methods: counting adds and multiplies gave 0.0054 petaflop/s-days, while GPU time at an assumed one-third utilisation gave 0.0058⁵. Both are a count of work done, not the GTX 580's peak rate⁵.

Why it matters

Training headlines quote enormous FLOP counts. Reading a count as if it were a chip's peak speed makes a training run look like a hardware spec, and the two numbers answer different questions.

Questions people ask

Are FLOPs and FLOPS the same thing?

No. FLOPs is a count of floating-point operations. FLOPS is a rate, operations per second. OpenAI's GPU-time method explicitly says it is not counting peak theoretical FLOPS, only an assumed fraction of that peak used to estimate operations actually performed⁵.

Why do training-FLOP formulas multiply by six?

Kaplan and colleagues count about 2N operations for a forward pass, because a multiply-accumulate is two operations, and about twice that again for the backward pass, which gives about 6N operations per token². Hoffmann and colleagues use the same 6ND shortcut³.

Does the EU AI Act threshold use FLOPs or chip speed?

Article 51 measures the cumulative computation used for training in floating-point operations, and presumes high-impact capabilities above 10 to the 25⁴. The threshold is that operation count⁴.

Related terms

Sources

Back to the glossary