# BERT

BERT is a transformer encoder pre-trained with masked language modelling that reads text in both directions. Learn how it works and where it is still used.

## Overview

BERT, short for Bidirectional Encoder Representations from Transformers, is a language model designed to understand text rather than to generate it. It is built from the encoder half of the [transformer](/tools/transformers). During pre-training, some words in a sentence are hidden and BERT learns to fill them in using the words on both sides. This forces it to build a rich, context-aware representation of every word.

After pre-training, BERT is fine-tuned for a specific task by adding a small output layer and training briefly on labelled data. With this recipe, one pre-trained model could be adapted to sentiment analysis, question answering, named entity recognition and many other tasks, often setting new records when it was released. BERT made "pre-train once, fine-tune many times" the standard approach in natural language processing.

Think of a crossword puzzle. To fill a blank, you look at the letters and clues on both sides, across and down. A model that reads only left to right sees half the clues. BERT reads in both directions at once, so it uses all the surrounding context to understand each word.

## What it is

| Part | Description |
|---|---|
| Tokeniser | WordPiece, splitting text into a vocabulary of about 30,000 sub-word pieces |
| Embeddings | Token, position and segment embeddings added together |
| Encoder stack | Transformer layers with bidirectional self-attention |
| [CLS] token | A special first token whose output summarises the whole input |
| [SEP] token | A separator between two segments, such as a question and a passage |
| [MASK] token | A placeholder for hidden words during pre-training |
| Task head | A small layer added on top for each fine-tuning task |

The original paper released two sizes.

| Model | Layers | Hidden size | Attention heads | Parameters |
|---|---|---|---|---|
| BERT-Base | 12 | 768 | 12 | about 110 million |
| BERT-Large | 24 | 1,024 | 16 | about 340 million |

Both accept inputs of up to 512 tokens.

## Where it came from

BERT was created by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google AI Language. The paper appeared on arXiv in October 2018 and was published at NAACL in 2019. Google released the code and pre-trained weights as open source in late 2018. The original models were pre-trained on BooksCorpus and English Wikipedia, about 3.3 billion words in total.

BERT built on earlier ideas. [Word embeddings](/tools/word-embeddings) such as word2vec gave each word a fixed vector. ELMo, in 2018, produced contextual vectors with recurrent networks. OpenAI's first GPT, also in 2018, pre-trained a transformer decoder that read left to right. BERT's contribution was deep bidirectional pre-training with a transformer encoder, made possible by the masked language modelling objective.

A wave of variants followed. In 2019 RoBERTa from Facebook AI trained longer on more data and dropped the next sentence task. DistilBERT from Hugging Face used [knowledge distillation](/tools/knowledge-distillation) to make a smaller, faster model. ALBERT shared parameters across layers. Later, ELECTRA and DeBERTa improved pre-training efficiency and attention. In October 2019 Google said it was using BERT in Search to better understand queries. In December 2024 Answer.AI and LightOn released ModernBERT, an updated encoder with a context length of 8,192 tokens and modern training techniques.

## How it works

### Step 1: Tokenise and build the input

Text is split into WordPiece tokens. A [CLS] token is added at the start and a [SEP] token at the end of each segment. Each token gets three embeddings: what it is, where it is, and which segment it belongs to. These are summed.

### Step 2: Encode with bidirectional attention

The sequence passes through the encoder layers. In each layer, every token attends to every other token, both before and after it, through [self-attention](/tools/attention-mechanism). There is no causal mask, unlike in the [GPT architecture](/tools/gpt-architecture).

### Step 3: Pre-train with masked language modelling

About 15 per cent of tokens are chosen for prediction. Of these, 80 per cent are replaced with [MASK], 10 per cent with a random token and 10 per cent are left unchanged. The model must predict the original token. The mixture stops the model relying on the [MASK] symbol, which never appears in real use.

### Step 4: Pre-train with next sentence prediction

The original BERT also learned to predict whether sentence B really followed sentence A. Later work, notably RoBERTa, found this task added little and dropped it.

### Step 5: Fine-tune for a task

A small head is added. For classification, a linear layer reads the [CLS] output. For token-level tasks such as named entity recognition, a layer reads each token's output. For extractive question answering, two layers predict the start and end of the answer span. The whole model is then trained for a few epochs on labelled examples.

A quick way to see masked prediction with Hugging Face Transformers:

```python
from transformers import pipeline

fill = pipeline("fill-mask", model="google-bert/bert-base-uncased")
print(fill("The capital of France is [MASK]."))
```

## Key concepts and terms

| Term | Meaning |
|---|---|
| Encoder | A transformer stack that builds representations of the input |
| Bidirectional | Using context from both left and right of each token |
| Masked language modelling | Predicting hidden tokens from their context |
| Next sentence prediction | Predicting whether two sentences are consecutive |
| WordPiece | A sub-word tokeniser that splits rare words into pieces |
| [CLS] representation | The output vector used to summarise the input |
| Fine-tuning | Training the pre-trained model further on a labelled task |
| Contextual embedding | A word vector that depends on surrounding words |
| Cross-encoder | Feeding two texts together into BERT to score their relation |
| Bi-encoder | Encoding texts separately and comparing their vectors |

## A worked example

An illustrative bank wants to route customer messages into 12 categories, such as card lost, mortgage question and fraud report. It has 18,000 labelled messages.

1. The team splits data into 15,000 training, 1,500 validation and 1,500 test messages.
2. They start from a BERT-Base checkpoint and add a classification head with 12 outputs on the [CLS] token.
3. Messages are tokenised and cut to 128 tokens, since most are short.
4. Fine-tuning runs for 3 epochs with a learning rate of 2e-5 and batch size 32. On one modern GPU this takes well under an hour.
5. On the test set, the model reaches about 93 per cent accuracy in this example, against about 84 per cent for a keyword-based classifier.
6. In production, one CPU server classifies each message in tens of milliseconds, with no per-request API cost.
7. For the fraud category, where errors are costly, messages with low confidence go to a person.

The figures are illustrative. The pattern, a pre-trained encoder fine-tuned on a few thousand labels, remains a strong, cheap baseline for classification.

## Strengths and limitations

**Strengths:**

- Strong at understanding tasks: classification, tagging and extractive question answering.
- Small and fast compared with large generative models.
- Cheap to fine-tune on modest labelled data.
- Many open checkpoints, including multilingual and domain-specific versions.
- Basis for many [embedding](/tools/embeddings) and reranking models.

**Limitations:**

- Not designed to generate fluent text.
- The original models handle only 512 tokens.
- Needs labelled data for each task, unlike prompting a generative model.
- Raw [CLS] or averaged outputs make weak sentence embeddings without further training.
- Older checkpoints were trained on older data and miss newer vocabulary.

## How it compares with alternatives

| Model | Type | Main strength | Main drawback |
|---|---|---|---|
| BERT | Encoder, bidirectional | Efficient understanding and fine-tuning | No generation, short context |
| RoBERTa or DeBERTa | Improved encoders | Higher accuracy on many benchmarks | Similar limits to BERT |
| ModernBERT | Modern encoder | Longer context, faster | Newer, fewer domain variants |
| [GPT-style decoders](/tools/gpt-architecture) | Decoder, left to right | Generation and zero-shot prompting | Larger and costlier |
| T5-style encoder-decoders | Both halves | Flexible text-to-text tasks | More complex to serve |

## Practical guidance and best practices

1. Use an encoder like BERT when the task is classification, tagging or ranking with clear labels.
2. Prefer newer encoders such as RoBERTa, DeBERTa or ModernBERT for best accuracy, and test against original BERT.
3. Pick a domain or multilingual checkpoint if your text is specialised or not in English.
4. Keep the maximum sequence length as short as your data allows to save compute.
5. Use small learning rates, around 1e-5 to 5e-5, and a few epochs; watch validation loss for overfitting.
6. For semantic search, use a model trained for sentence embeddings rather than raw BERT outputs.
7. Use [cross-encoders](/tools/cross-encoder-reranking) built on BERT for reranking a short list of results.
8. Distil or quantise for low-latency deployment.

## Common mistakes

- Using raw BERT outputs as sentence embeddings and getting poor search results.
- Truncating long documents silently and losing the key part.
- Fine-tuning with a high learning rate and destroying pre-trained knowledge.
- Using an uncased model where capitalisation matters, such as names.
- Comparing BERT with large chat models on generation tasks it was never built for.

## Frequently asked questions

### Is BERT still used?

Yes. Encoder models in the BERT family remain widely used for classification, entity recognition, embeddings and reranking, because they are fast and cheap. Newer encoders have improved on the original, but the design lives on.

### What is the difference between BERT and GPT?

BERT is an encoder that reads text in both directions and is trained to fill in masked words. GPT is a decoder that reads left to right and is trained to predict the next word. BERT is better suited to understanding; GPT to generation.

### What does "uncased" mean?

An uncased model lowercases text and removes accent marks before tokenising. A cased model keeps capitalisation. Cased models are better when case carries meaning.

### Can BERT handle long documents?

The original handles 512 tokens. For longer text you can split it into chunks, use long-context variants, or use newer encoders such as ModernBERT with longer limits.

### How much labelled data do I need?

It depends on the task. A few thousand examples often give strong results for classification, and a few hundred can work for simple tasks. More data and balanced classes help.

### Is BERT open source?

Google released the original code and weights under the Apache 2.0 licence. Many derived checkpoints are published on [Hugging Face](/tools/hugging-face) with their own licences.

## Glossary

- Transformer: a neural network architecture based on attention.
- Self-attention: each token weighing every other token to build its representation.
- Token: a sub-word unit produced by the tokeniser.
- Checkpoint: a saved set of model weights.
- Epoch: one pass over the training data.
- Head: a small task-specific layer on top of the encoder.
- Span: a continuous stretch of text, such as an answer in a passage.
- Distillation: training a smaller model to mimic a larger one.

## Summary

BERT is a transformer encoder from Google, released in 2018, that learns deep bidirectional representations by predicting masked words. Fine-tuned with a small task head, it set new standards across language understanding tasks and made pre-training followed by fine-tuning the norm. It does not generate text and the original handles only 512 tokens, but its family, including RoBERTa, DeBERTa, DistilBERT and ModernBERT, remains a fast, inexpensive choice for classification, tagging, embeddings and reranking.