Activation functions: ReLU, GELU, sigmoid, tanh and more
The activation is the bend that makes depth worth having. Compare twelve of them, from the original step function to GELU in transformers, and see why their slopes matter as much as their shapes.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The solid curve is the activation f(z); the dashed curve is its slope f′(z), the number backpropagation multiplies by. The white point and tangent show both at the z you choose.
Things to try
- Compare sigmoid and ReLU at z = 4: the sigmoid slope is almost zero, which is the vanishing gradient problem.
- Look at ReLU for negative z: slope zero, so a neuron stuck there stops learning. Leaky ReLU fixes it.
- Show all curves: GELU, Swish and Mish are smooth versions of ReLU.
- Note that only step has a slope of zero everywhere: it cannot be trained with gradients.
How it works in real systems
Most deep networks use ReLU or a smooth relative. Transformers such as BERT and GPT use GELU, many recent language models use SwiGLU (Swish with a gate), and sigmoid or softmax appear at the output to turn scores into probabilities.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · All of them in PyTorch
import torch, torch.nn.functional as F
x = torch.linspace(-4, 4, 9)
acts = {"sigmoid": torch.sigmoid, "tanh": torch.tanh, "relu": F.relu, "leaky_relu": lambda t: F.leaky_relu(t, 0.01),
"elu": F.elu, "selu": F.selu, "gelu": F.gelu, "silu (swish)": F.silu, "mish": F.mish, "softplus": F.softplus}
for name, f in acts.items():
print(f"{name:13s}", f(x).numpy().round(3))
Step 2 · Their derivatives, with autograd
The derivative decides how gradients flow back. Sigmoid's is at most 0.25, which is why deep sigmoid networks suffer vanishing gradients.
for name, f in acts.items():
t = x.clone().requires_grad_()
f(t).sum().backward()
print(f"{name:13s}", t.grad.numpy().round(3))
Step 3 · Which to use
Transformers use GELU or SwiGLU (a gated SiLU) in their MLP blocks; convolutional networks mostly ReLU or SiLU; outputs use sigmoid (binary), softmax (multi-class) or nothing (regression).
class SwiGLU(torch.nn.Module): # the MLP block of Llama-style models
def __init__(self, d, hidden):
super().__init__()
self.w1 = torch.nn.Linear(d, hidden, bias=False)
self.w3 = torch.nn.Linear(d, hidden, bias=False)
self.w2 = torch.nn.Linear(hidden, d, bias=False)
def forward(self, x):
return self.w2(F.silu(self.w1(x)) * self.w3(x))