Activation functions

Overview

An activation function is the small non-linear step inside every neuron of a neural network. A neuron first computes a weighted sum of its inputs plus a bias. On its own that is a straight-line (linear) operation. The activation function then bends the result, for example by setting negative values to zero. That bend is what lets a stack of layers represent curved, complicated relationships. Without it, any number of layers would collapse into one big linear function, no more powerful than linear regression.

An everyday picture is a dimmer switch compared with a plain on/off switch. A plain switch (the step function) is either fully off or fully on. A dimmer lets the light rise smoothly. Some dimmers stay dark until you pass a certain point and then brighten steadily, which is roughly what ReLU does. Different activation functions are different dimmer designs, and the choice affects how easily the whole lighting system (the network) can be tuned.

This page walks through the main functions, from the historical step and sigmoid to ReLU and its variants, GELU and SiLU, softmax for outputs, and the gated units (GLU, SwiGLU) used in modern language models. It explains vanishing gradients and dead ReLUs, and which architectures use which function.

What it is

An activation function f maps a single number (the pre-activation z) to an output a = f(z). It is usually applied element-wise to a whole layer. The properties that matter are below.

The table of main functions:

Where it came from

Warren McCulloch and Walter Pitts described a threshold neuron in 1943: it outputs 1 if its weighted input passes a threshold, otherwise 0. Frank Rosenblatt's perceptron of 1958 used the same step function. The step has a zero gradient almost everywhere, so it cannot be trained by gradient descent.

The backpropagation era of the 1980s used smooth, S-shaped functions: the logistic sigmoid and tanh. They work, but their gradients become tiny when inputs are large in magnitude, which made deep networks hard to train.

In 2010 Vinod Nair and Geoffrey Hinton used rectified linear units in restricted Boltzmann machines (ICML 2010). In 2011 Xavier Glorot, Antoine Bordes and Yoshua Bengio showed that rectifiers helped deep supervised networks. AlexNet in 2012 used ReLU and helped make it the default. Andrew Maas, Awni Hannun and Andrew Ng proposed the leaky ReLU in 2013. Kaiming He and colleagues introduced PReLU, with a learnable negative slope, in 2015. Djork-Arné Clevert, Thomas Unterthiner and Sepp Hochreiter proposed ELU in 2015, and Günter Klambauer and colleagues introduced SELU and self-normalising networks in 2017.

Dan Hendrycks and Kevin Gimpel proposed GELU in 2016; the same paper also described the function later known as SiLU. Stefan Elfwing, Eiji Uchibe and Kenji Doya proposed it again as the sigmoid-weighted linear unit in 2017 for reinforcement learning. Prajit Ramachandran, Barret Zoph and Quoc Le found it by automated search later in 2017 and named it Swish. On the gating side, Yann Dauphin and colleagues introduced the gated linear unit (GLU) in "Language Modeling with Gated Convolutional Networks" (2016), and Noam Shazeer's 2020 paper "GLU Variants Improve Transformer" showed that variants such as SwiGLU and GEGLU improve Transformer feed-forward layers. LLaMA (2023) is one of the well-known models that adopted SwiGLU.

How it works

The formulas

step(z)     = 1 if z ≥ 0 else 0
sigmoid(z)  = σ(z) = 1 / (1 + e^(−z))          σ'(z) = σ(z)(1 − σ(z)) ≤ 0.25
tanh(z)     = (e^z − e^(−z)) / (e^z + e^(−z))   tanh'(z) = 1 − tanh(z)² ≤ 1
ReLU(z)     = max(0, z)
LeakyReLU(z)= z if z > 0 else a · z
ELU(z)      = z if z > 0 else α (e^z − 1)
SELU(z)     = λ · ELU_α(z),  α ≈ 1.6733, λ ≈ 1.0507
GELU(z)     = z · Φ(z)
SiLU(z)     = z · σ(z)              (Swish with β = 1)
softmax(z)_k= e^(z_k) / Σ_j e^(z_j)
GLU(x)      = (x W + b) ⊙ σ(x V + c)
SwiGLU(x)   = SiLU(x W) ⊙ (x V)

Here z is the pre-activation, e is Euler's number, a is the small leaky slope (0.01 by default in PyTorch's nn.LeakyReLU), α and λ are fixed constants for ELU and SELU, Φ is the standard normal cumulative distribution function, β is Swish's optional slope parameter, z_k is the score for class k, x is the layer input, W, V, b, c are learned weights and biases, and ⊙ means element-wise multiplication.

Vanishing gradients

During backpropagation, the gradient that reaches an early layer is a product of many factors, one per layer, and each includes the activation's derivative. The sigmoid's derivative is at most 0.25. Ten sigmoid layers can multiply the gradient by at most 0.25¹⁰, roughly one in a million, before weights are even counted. Early layers then barely learn. Tanh is better (its derivative reaches 1 at zero) but still saturates for large inputs. ReLU's derivative is exactly 1 for positive inputs, so it passes gradients through unchanged, which is the main reason it made deep networks practical. Good weight initialization, normalization layers and residual connections attack the same problem from other sides.

Dead ReLUs

ReLU has a weakness. If a neuron's input is negative for every example, its output and its gradient are both zero, so its weights never change again. The neuron is "dead". This can happen after a large update caused by a high learning rate, or with a bad bias initialisation. Leaky ReLU and PReLU keep a small slope for negative inputs so some gradient always flows. ELU, GELU and SiLU are smooth near zero and let small negative values through, which also helps.

Smooth modern functions

GELU weights each input by the probability that a standard normal variable is below it. Large positive inputs pass almost unchanged, large negative inputs go to nearly zero, and inputs near zero are partly let through. SiLU, z · σ(z), has a similar shape. Both are non-monotonic: they dip slightly below zero for negative inputs before returning towards zero. In practice they tend to give small but consistent gains over ReLU in Transformers and some vision models, at a little more compute. PyTorch's nn.GELU also offers a faster tanh-based approximation with approximate='tanh'.

Softmax for outputs

Softmax turns a vector of scores into probabilities that sum to 1. It is used at the output of multi-class classifiers and inside the attention mechanism. Because it exponentiates, implementations subtract the maximum score first for numerical stability. In PyTorch you normally give raw scores (logits) to nn.CrossEntropyLoss, which combines log-softmax and the loss in one stable step (see loss functions).

Gated units

A gated linear unit splits a layer into two linear projections. One is the "value"; the other goes through a sigmoid and acts as a gate between 0 and 1 that decides how much of each value passes. This is the same idea as the gates in LSTM and GRU cells, applied inside a feed-forward block. SwiGLU replaces the sigmoid gate with SiLU. A SwiGLU block has three weight matrices instead of two, so models often shrink the hidden size by about a third to keep the parameter count comparable.

Which architectures use which

• Classic MLPs and CNNs such as AlexNet and ResNet: ReLU in hidden layers.
• Recurrent networks: tanh for the hidden state and sigmoid for gates.
• BERT and GPT-2-style Transformers: GELU in the feed-forward block.
• EfficientNet: Swish/SiLU.
• Many recent large language models, including LLaMA: SwiGLU.
• Output layers: sigmoid for binary or multi-label tasks, softmax for single-label multi-class, no activation (identity) for regression.

import torch
from torch import nn

x = torch.linspace(-3, 3, 7)
for act in [nn.ReLU(), nn.LeakyReLU(0.01), nn.GELU(), nn.SiLU(), nn.Tanh()]:
    print(type(act).__name__, act(x).round(decimals=2))

class SwiGLU(nn.Module):
    def __init__(self, d_model, d_hidden):
        super().__init__()
        self.w = nn.Linear(d_model, d_hidden, bias=False)
        self.v = nn.Linear(d_model, d_hidden, bias=False)
        self.out = nn.Linear(d_hidden, d_model, bias=False)
    def forward(self, x):
        return self.out(nn.functional.silu(self.w(x)) * self.v(x))

Key concepts and terms

A worked example

Take one hidden layer with three neurons whose pre-activations for an input are −2, 0.5 and 3. The figures are illustrative.

• ReLU gives 0, 0.5 and 3. Gradients are 0, 1 and 1. The first neuron passes no gradient back for this input.
• Leaky ReLU (0.01) gives −0.02, 0.5 and 3. Gradients are 0.01, 1 and 1. The first neuron can still learn, slowly.
• Sigmoid gives about 0.119, 0.622 and 0.953. Gradients are 0.105, 0.235 and 0.045. Even the strongest input passes only a small gradient.
• GELU gives about −0.045, 0.346 and 2.996. Gradients are about −0.085, 0.867 and 1.012. Near zero GELU is softer than ReLU.

Now stack ten layers and follow one path through them. With sigmoid at its best case, each layer multiplies the gradient by at most 0.25, so ten layers give at most about 0.00000095. With ReLU on an active path, each factor is 1, so the activation does not shrink the gradient at all. That difference is why deep sigmoid networks stalled and deep ReLU networks trained.

Finally, an output layer for three classes produces logits 2.0, 1.0 and 0.1. Softmax exponentiates them to about 7.39, 2.72 and 1.11, which sum to 11.21, so the probabilities are about 0.66, 0.24 and 0.10.

Strengths and limitations

Strengths:

• Non-linear activations give deep networks their expressive power.
• ReLU is very cheap and largely solves vanishing gradients for active units.
• Smooth functions such as GELU and SiLU often improve accuracy in Transformers.
• Gated units add input-dependent control and are a proven improvement in large language models.
• Output activations (sigmoid, softmax) map scores to well-defined probabilities.

Limitations:

• Sigmoid and tanh saturate, causing vanishing gradients in deep stacks.
• ReLU units can die, especially with high learning rates.
• Smooth functions cost more to compute than ReLU.
• SELU's self-normalising property relies on specific initialisation and dropout variants.
• Gated units add parameters and memory; hidden sizes must be adjusted to compare fairly.

How it compares with alternatives

Practical guidance and best practices

1. Use ReLU as a reliable default for MLPs and CNNs; switch to GELU or SiLU if you are building a Transformer or want a small accuracy gain.
2. Match initialisation to the activation: He (Kaiming) for ReLU family, Xavier (Glorot) for tanh and sigmoid. See weight initialization.
3. Do not put an activation on a regression output; leave it linear.
4. Use sigmoid outputs for multi-label tasks and softmax for single-label multi-class tasks.
5. Pass logits to the loss rather than applying sigmoid or softmax yourself.
6. If many ReLU units are dead (always zero), lower the learning rate or try Leaky ReLU.
7. When using SwiGLU, reduce the hidden size by about one third to keep parameters comparable with a standard feed-forward block.
8. Combine activations with normalization layers and residual connections for very deep models.

Common mistakes

• Building a deep network with no activations between layers, which reduces it to a single linear map.
• Applying softmax twice: once in the model and again inside nn.CrossEntropyLoss.
• Using sigmoid in hidden layers of a deep network and wondering why early layers do not learn.
• Using ReLU on the output of a regression that can be negative.
• Using SELU without its matching initialisation and expecting self-normalisation.
• Treating the activation as the main lever for accuracy; data, architecture and tuning usually matter more.

Frequently asked questions

Why do neural networks need activation functions?

Because composing linear functions gives another linear function. Without non-linear activations, a network with a hundred layers can represent only what one linear layer can. The activation lets each layer bend the representation so that depth adds power.

Is ReLU still the best default?

For convolutional networks and simple MLPs, ReLU remains a strong, cheap default. For Transformers, GELU or a gated variant such as SwiGLU is now more common. The differences are usually modest, so pick the standard for your architecture and focus tuning effort elsewhere.

What is the difference between SiLU and Swish?

They are the same function, z · σ(z), when Swish's β parameter is 1. The function was described by Hendrycks and Gimpel in 2016, named SiLU by Elfwing and colleagues in 2017, and named Swish by Ramachandran and colleagues later in 2017. PyTorch calls it nn.SiLU.

What causes the vanishing gradient problem?

Gradients in early layers are products of many per-layer factors. If those factors are below 1, as with saturated sigmoid or tanh units, the product shrinks exponentially with depth. ReLU-family activations, careful initialisation, normalisation and residual connections all reduce the problem.

Why is softmax used in attention?

Attention needs weights that are positive and sum to 1, so that the output is a weighted average of value vectors. Softmax turns arbitrary similarity scores into exactly such weights, while keeping the operation differentiable.

Can I learn the activation function itself?

Partly. PReLU learns the negative slope per channel, and Swish can learn its β. Fully learned activations have been studied, but fixed, well-understood functions remain the norm in practice.

Glossary

• Activation function: a non-linear function applied to a neuron's weighted input.
• ReLU: rectified linear unit, max(0, z).
• Leaky ReLU: ReLU with a small slope for negative inputs.
• GELU: Gaussian error linear unit, z · Φ(z).
• SiLU: sigmoid linear unit, z · σ(z), also called Swish.
• Softmax: a function turning scores into probabilities that sum to one.
• GLU: gated linear unit, a value multiplied by a sigmoid gate.
• SwiGLU: a gated unit using SiLU as the gate.
• Saturation: the flat region of a function where gradients vanish.
• Dead neuron: a unit that outputs zero for all inputs and stops learning.

Summary

Activation functions supply the non-linearity that makes deep networks more than linear models. The step function started the field but cannot be trained by gradients. Sigmoid and tanh made backpropagation possible but saturate, causing vanishing gradients in deep stacks. ReLU fixed much of that cheaply and became the default for CNNs and MLPs, with Leaky ReLU, PReLU, ELU and SELU addressing dead units and normalisation. Smooth functions such as GELU and SiLU now dominate Transformers, softmax turns scores into probabilities for outputs and attention, and gated units such as SwiGLU power the feed-forward blocks of many large language models. Pick the standard for your architecture, pair it with the right initialisation, and spend most tuning effort elsewhere.

Related

It belongs to Neural network basics in my stack, under Deep learning. Others in the same group: