# What is Activation functions? Definition and meaning

An activation function is the non-linear function applied to a neuron's weighted sum before its output is passed on.

## Definition

**Activation functions**: An activation function is the non-linear function applied to a neuron's weighted sum before its output is passed on.

In short: The bend that makes depth.

## When it is used

Every hidden layer of a neural network uses one, from small MLPs to large Transformers.

## Where it fits

Activation functions belongs to **Neural network basics** in the [Deep learning](/topics/deep-learning) topic. These are the other terms in the same group:

- **[Neural networks](/glossary/neural-networks)**: A neural network is a model made of layers of simple units that each compute a weighted sum and a non-linear function, with the weights learned from data.
- **[Weight initialization](/glossary/weight-initialization)**: Weight initialization is the choice of starting values for a neural network's weights and biases before training begins.
- **[Learning rate schedules](/glossary/learning-rate-schedules)**: A learning rate schedule is a rule that changes the optimiser's learning rate as training progresses.

## Learn more

The full guide to **[Activation functions](/tools/activation-functions)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [A3C](/glossary/a3c) · Next term: [Active learning](/glossary/active-learning)
