Glossary · Deep learning
What is Transformers? Definition and meaning
Neural networks built on self-attention, letting every element of the input weigh every other element.
Definition
Transformers: Neural networks built on self-attention, letting every element of the input weigh every other element.
In short: Attention-based networks for sequences and sets.
When it is used
The backbone of language models, vision transformers and tabular foundation models.
Where it fits
Transformers belongs to Architectures in the Deep learning topic. These are the other terms in the same group:
- CNN: Networks that apply learned filters across local neighbourhoods, sharing weights over positions.
- Recurrent neural networks: A neural network that processes a sequence one step at a time, updating a hidden state that summarises everything seen so far.
- LSTM: A recurrent network with gates that control what is stored, forgotten and emitted over time.
- GRU: A recurrent unit with fewer gates than an LSTM, often similarly accurate and faster to train.
- ResNet: A convolutional architecture whose residual connections make very deep networks trainable.
- ViT: Splits an image into patches and processes them as a sequence with a transformer.
- Autoencoders: A network that encodes an input into a smaller code and decodes it back, learning what matters.
- VAE: An autoencoder that learns a distribution over a latent space and is trained with a variational objective.
- GAN: Two networks compete: a generator produces samples and a discriminator tries to tell them from real data.
- Diffusion models: Generative models that learn to denoise data step by step, turning noise into a sample.
Learn more
The full guide to Transformers explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.
Previous term: Transfer learning · Next term: Tree-of-thoughts