# Activation functions: ReLU, GELU, sigmoid, tanh and more

The activation is the bend that makes depth worth having. Compare twelve of them, from the original step function to GELU in transformers, and see why their slopes matter as much as their shapes.

<div data-sim="act"><p>Interactive simulation. It needs JavaScript; the text below explains what it shows.</p></div>

## What you are looking at

The solid curve is the activation f(z); the dashed curve is its slope f′(z), the number backpropagation multiplies by. The white point and tangent show both at the z you choose.

## Things to try

- Compare sigmoid and ReLU at z = 4: the sigmoid slope is almost zero, which is the vanishing gradient problem.
- Look at ReLU for negative z: slope zero, so a neuron stuck there stops learning. Leaky ReLU fixes it.
- Show all curves: GELU, Swish and Mish are smooth versions of ReLU.
- Note that only step has a slope of zero everywhere: it cannot be trained with gradients.

## How it works in real systems

Most deep networks use ReLU or a smooth relative. Transformers such as BERT and GPT use GELU, many recent language models use SwiGLU (Swish with a gate), and sigmoid or softmax appear at the output to turn scores into probabilities.

## Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

<div class="cl-start"></div>

### Step 1 · All of them in PyTorch

```python
import torch, torch.nn.functional as F
x = torch.linspace(-4, 4, 9)
acts = {"sigmoid": torch.sigmoid, "tanh": torch.tanh, "relu": F.relu, "leaky_relu": lambda t: F.leaky_relu(t, 0.01),
        "elu": F.elu, "selu": F.selu, "gelu": F.gelu, "silu (swish)": F.silu, "mish": F.mish, "softplus": F.softplus}
for name, f in acts.items():
    print(f"{name:13s}", f(x).numpy().round(3))
```

### Step 2 · Their derivatives, with autograd

The derivative decides how gradients flow back. Sigmoid's is at most 0.25, which is why deep sigmoid networks suffer vanishing gradients.

```python
for name, f in acts.items():
    t = x.clone().requires_grad_()
    f(t).sum().backward()
    print(f"{name:13s}", t.grad.numpy().round(3))
```

### Step 3 · Which to use

Transformers use GELU or SwiGLU (a gated SiLU) in their MLP blocks; convolutional networks mostly ReLU or SiLU; outputs use sigmoid (binary), softmax (multi-class) or nothing (regression).

```python
class SwiGLU(torch.nn.Module):            # the MLP block of Llama-style models
    def __init__(self, d, hidden):
        super().__init__()
        self.w1 = torch.nn.Linear(d, hidden, bias=False)
        self.w3 = torch.nn.Linear(d, hidden, bias=False)
        self.w2 = torch.nn.Linear(hidden, d, bias=False)
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))
```

<div class="cl-end"></div>

## More simulations

[← One neuron](/ai-simulations/one-neuron) · [All AI Simulations](/ai-simulations) · [Loss functions →](/ai-simulations/loss-functions)
