Glossary · Deep learning
What is Activation functions? Definition and meaning
An activation function is the non-linear function applied to a neuron's weighted sum before its output is passed on.
Definition
Activation functions: An activation function is the non-linear function applied to a neuron's weighted sum before its output is passed on.
In short: The bend that makes depth.
When it is used
Every hidden layer of a neural network uses one, from small MLPs to large Transformers.
Where it fits
Activation functions belongs to Neural network basics in the Deep learning topic. These are the other terms in the same group:
- Neural networks: A neural network is a model made of layers of simple units that each compute a weighted sum and a non-linear function, with the weights learned from data.
- Weight initialization: Weight initialization is the choice of starting values for a neural network's weights and biases before training begins.
- Learning rate schedules: A learning rate schedule is a rule that changes the optimiser's learning rate as training progresses.
Learn more
The full guide to Activation functions explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: A3C · Next term: Active learning