Activation function

In short

An activation function is the fixed rule a unit in a neural network uses to turn its inputs into an output. It is what stops a stack of layers from collapsing into one straight-line calculation.

Also called: Nonlinearity
Topic: How AI models work
Level: Beginner
First described: Warren McCulloch and Walter Pitts, 1943, in "A logical calculus of the ideas immanent in nervous activity", which treats neural events as all-or-none.¹

In 1943 Warren McCulloch and Walter Pitts treated the all-or-none character of nervous activity in logic: for their purposes a neural event is on or off.¹ Hendrycks and Gimpel, in their survey of activation functions, describe those early units as binary thresholds, then the smoothing of that hard step into a sigmoid. A sigmoid gives the unit a graded output and lets it be trained by backpropagation. They add that, as networks got deeper, sigmoids trained less well than the rectified linear unit.²

The rectified linear unit, or ReLU, is max(0, x). If the input is positive, the output equals the input. If it is negative, the output is zero. Vinod Nair and Geoffrey Hinton used these units in 2010. Unlike binary units, they wrote, rectified linear units preserve information about relative intensities as it passes through several layers of feature detectors.³ Xavier Glorot, Antoine Bordes and Yoshua Bengio then tested the rectifier, which they also write as max(0, x), as a replacement for the hyperbolic tangent and the sigmoid in deep supervised networks.⁴

Other functions have been proposed for the same job. Hendrycks and Gimpel proposed the Gaussian Error Linear Unit, or GELU, equal to x times the standard Gaussian cumulative distribution function. A ReLU keeps or zeroes its input according to the sign. A GELU weights the input by its value instead. On the vision, language and speech tasks they tried, GELU beat ReLU and ELU.² The original transformer uses a ReLU inside each feed-forward block: two linear maps with max(0, xW + b) between them.⁵

An activation function is not a weight. Weights are the numbers training changes. The activation is the rule applied afterwards, chosen by whoever designs the network. Hendrycks and Gimpel note that the choice matters because, without a nonlinear activation, the network is only a deep linear classifier however many layers you add.²

Example

If a unit's combined input is 2, a ReLU outputs 2. If the combined input is -3, a ReLU outputs 0. The rule is max(0, x), and nothing more.³

Why it matters

Without a nonlinear activation, stacking layers would not add a new kind of pattern. Which function you pick also changes how readily a deep network trains.²⁴

Questions people ask

What is a ReLU?

ReLU means rectified linear unit. Its output is max(0, x): the input itself when that input is positive, and zero when it is negative. Nair and Hinton set the name out in 2010.³

Why not use a straight line as the activation?

Hendrycks and Gimpel note that without a nonlinear activation, a deep network is only a linear classifier, however many layers it has. The layers collapse into one weighted sum.²

Is GELU the same as ReLU?

No. A ReLU keeps or drops its input based on the sign. A GELU, proposed by Hendrycks and Gimpel, multiplies the input by a value from the Gaussian cumulative distribution, so the gate is soft rather than on or off.²

Related terms

Sources

Back to the glossary