GPT architecture
The GPT architecture is a decoder-only transformer trained to predict the next token. Learn its layers, causal attention, pre-training and why it scales so well.
Overview
The GPT architecture is the design behind most of today's large language models. GPT stands for Generative Pre-trained Transformer. It is a decoder-only transformer: a stack of identical layers that reads a sequence of tokens and predicts the next one. Each token can look only at the tokens before it, never after. The model generates text by predicting one token, appending it to the input, and repeating.
The training goal is simple: given all the text so far, guess the next token. Trained on enormous amounts of text and code, this one objective teaches grammar, facts, styles, reasoning patterns and programming. Instruction tuning and preference training then turn the raw predictor into a helpful assistant. The same basic design, with refinements, scales from millions to hundreds of billions of parameters.
Think of a skilled storyteller continuing a tale one word at a time. At each moment they recall everything said so far, weigh which earlier parts matter now, and choose the next word. They cannot peek at words they have not said yet. A GPT model is that storyteller, built from layers of attention and arithmetic.
What it is
| Component | Role |
|---|---|
| Tokeniser | Splits text into tokens, often with byte-level BPE |
| Token embedding | Turns each token ID into a vector |
| Position information | Tells the model where each token sits, by learned positions or rotary encoding |
| Decoder blocks | Repeated layers of causal self-attention and a feed-forward network |
| Causal mask | Stops each position from attending to later positions |
| Residual connections | Add each sub-layer's output back to its input, easing training |
| Layer normalisation | Keeps activations in a stable range; see normalisation layers |
| Output head | Maps the final vector to scores over the whole vocabulary |
Compared with BERT, the key difference is the causal mask. BERT sees both sides and fills in blanks. GPT sees only the past and predicts the future, which is exactly what generation needs.
Where it came from
The transformer was introduced by Google researchers in the 2017 paper "Attention Is All You Need", with an encoder and a decoder for translation. In June 2018 OpenAI published "Improving Language Understanding by Generative Pre-Training", introducing GPT, often called GPT-1. It used only a decoder stack of 12 layers, about 117 million parameters, pre-trained on books and then fine-tuned for tasks.
GPT-2 followed in February 2019, with up to 1.5 billion parameters trained on a large web-text dataset. It moved layer normalisation to the input of each sub-layer, a change that made deep stacks easier to train, and showed that a large language model could perform tasks without task-specific fine-tuning. OpenAI released it in stages over 2019. In May 2020 GPT-3 scaled the design to 175 billion parameters and showed strong few-shot learning from prompts alone.
Research on scaling laws, notably by OpenAI researchers in 2020 and DeepMind's Chinchilla work in 2022, described how loss falls predictably as model size, data and compute grow, and that many models had been trained on too little data for their size. Since then the decoder-only design has become the default for large language models across labs. Common refinements include rotary position embeddings, RMSNorm, gated feed-forward layers such as SwiGLU, grouped-query attention to shrink the KV cache, and mixture-of-experts layers that activate only part of the network per token.
How it works
Step 1: Tokenise and embed
Text becomes a list of token IDs. Each ID is looked up in an embedding table to get a vector. Position information is added or applied inside attention.
Step 2: Causal self-attention
In each block, every token produces a query, a key and a value. Each token compares its query with the keys of itself and all earlier tokens, turns the scores into weights with softmax, and takes a weighted mix of their values. The causal mask sets scores for future tokens to minus infinity so they get zero weight. Several heads do this in parallel, each free to track different relationships. See attention mechanism.
In PyTorch the core operation can be written in one call:
import torch.nn.functional as F
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
Step 3: Feed-forward network
Each token's vector then passes through a small two-layer network applied independently at each position. This is where much of the model's stored knowledge is thought to live, and where most parameters sit.
Step 4: Residuals and normalisation
Each sub-layer's output is added to its input, and inputs are normalised before each sub-layer. This keeps signals stable through dozens of layers.
Step 5: Predict the next token
After the last block, the output head turns the final vector at each position into a score for every vocabulary token. Softmax turns scores into probabilities.
Step 6: Train on next-token prediction
During pre-training, the model predicts every position's next token in parallel across huge batches, and the loss is cross-entropy against the real next token. Training uses backpropagation and optimisers such as AdamW over trillions of tokens.
Step 7: Align and generate
Pre-trained models are then tuned with supervised fine-tuning and preference methods such as RLHF or DPO. At inference the model generates one token at a time, reusing past keys and values from the KV cache, and a sampling rule picks each token.
Key concepts and terms
| Term | Meaning |
|---|---|
| Decoder-only | A transformer with causal attention and no separate encoder |
| Autoregressive | Generating each token based on those before it |
| Causal mask | The rule that hides future positions |
| Context length | Maximum number of tokens the model can attend to |
| Logits | Raw scores over the vocabulary before softmax |
| Cross-entropy loss | The training measure of next-token prediction error |
| Residual stream | The running vector each layer reads from and adds to |
| Rotary position embedding | A way to encode position by rotating query and key vectors |
| Scaling laws | Empirical rules linking loss to model size, data and compute |
| Temperature | A setting that sharpens or flattens the output distribution |
A worked example
Here is an illustrative small GPT-style model, sized for learning purposes.
- Vocabulary: 32,000 tokens. Model width: 768. Layers: 12. Attention heads: 12, each of size 64. Context length: 1,024.
- Embedding table: 32,000 × 768, about 24.6 million parameters.
- Each block has attention weights of about 4 × 768 × 768, roughly 2.4 million, and a feed-forward network with an inner size of 3,072, about 2 × 768 × 3,072, roughly 4.7 million. Total per block: about 7.1 million.
- Twelve blocks: about 85 million. With embeddings, about 110 million parameters, if the output head shares weights with the embedding table.
Now trace one prediction. The prompt is "The cat sat on the". It becomes 5 tokens. Each passes through all 12 blocks. At the last position, the model's output head gives probabilities such as "mat" 0.31, "floor" 0.18, "sofa" 0.09. With a low temperature, "mat" is chosen. The new token is appended, its keys and values are added to the cache, and the next step needs computation only for that one token.
Training such a model on a few billion tokens is feasible on a small number of GPUs. Production models use the same structure with far more layers, width and data.
Strengths and limitations
Strengths:
- One simple objective scales to very broad abilities.
- Naturally suited to generation, chat and code.
- Training is highly parallel across positions on GPUs.
- Prompting and in-context learning remove the need for task-specific heads.
- A large ecosystem of tools, kernels and serving engines.
Limitations:
- Generation is sequential, one token at a time.
- Attention cost grows with the square of context length in standard form.
- The KV cache makes long contexts memory-hungry.
- Next-token training does not guarantee factual accuracy.
- Very large models are costly to train and serve.
How it compares with alternatives
| Architecture | How it processes text | Best for | Main trade-off |
|---|---|---|---|
| Decoder-only (GPT) | Causal, left to right | Generation, general assistants | Sequential decoding |
| Encoder-only (BERT) | Bidirectional | Classification, embeddings | Cannot generate fluently |
| Encoder-decoder (T5-style) | Encoder reads, decoder writes | Translation, summarisation | More complex |
| State space models (Mamba) | Recurrent-style fixed state | Very long sequences | Less mature ecosystem |
| Mixture of experts decoders | Routes tokens to sub-networks | More capacity per unit of compute | Routing and memory complexity |
Practical guidance and best practices
- When studying the design, build a small GPT yourself; a few hundred lines of code show every part.
- Check the context length, attention type and parameter count of any model before planning hardware.
- Use efficient kernels such as FlashAttention for training and inference.
- Follow compute-optimal guidance from scaling research when choosing model size versus data.
- Prefer grouped-query attention or similar for models meant for long-context serving.
- Keep tokeniser and model versions paired; a mismatch breaks everything.
- Evaluate generation quality with task-specific tests, not only training loss.
Common mistakes
- Thinking GPT is a single model; it is a family of designs and a general architecture.
- Forgetting the causal mask when implementing attention, letting the model cheat during training.
- Assuming more parameters always help without enough training data.
- Confusing the context window with long-term memory.
- Judging a raw pre-trained model as if it were an instruction-tuned assistant.
Frequently asked questions
Why is it called decoder-only?
The original transformer had an encoder and a decoder. GPT keeps only the decoder-style stack with causal attention and drops the encoder and the cross-attention that linked them.
Why does next-token prediction lead to such broad abilities?
To predict text well across many sources, a model must capture grammar, facts, styles and patterns of reasoning. At large scale, these become usable abilities, although they remain imperfect and are shaped further by fine-tuning.
What changed between GPT-2 and modern models?
The core structure is the same. Common changes include rotary positions, RMSNorm, gated feed-forward layers, grouped-query attention, mixture-of-experts layers, longer context and far more data and compute, plus extensive alignment training.
How does the model know word order?
Through position information. Early GPT models learned a vector for each position. Many current models use rotary position embeddings, which encode relative position inside attention.
Is every large language model a GPT?
Most widely used large language models use a decoder-only transformer, so they share the GPT architecture in a broad sense. Some use other designs, such as hybrids with state space layers.
What limits the context length?
Attention cost and KV cache memory grow with length, and models must be trained on long sequences to use them well. Efficient attention kernels and position schemes help extend it.
Glossary
- Token: a unit of text, often a word piece.
- Embedding: a learned vector for a token.
- Attention head: one of several parallel attention computations.
- Feed-forward network: a small per-position neural network inside each block.
- Softmax: a function that turns scores into probabilities.
- Pre-training: the large first stage of learning from raw text.
- Fine-tuning: later training for specific behaviour.
- Sampling: choosing each output token from the predicted probabilities.
Summary
The GPT architecture is a decoder-only transformer that uses causal self-attention to predict the next token, generating text one token at a time. Introduced by OpenAI in 2018 and scaled through GPT-2 and GPT-3, it showed that one simple training objective, applied to vast data, yields broad language and coding abilities. Modern models keep the same core and add refinements such as rotary positions, RMSNorm, grouped-query attention and mixture-of-experts layers. Its main costs are sequential generation and memory for long contexts, which serving techniques work hard to reduce.
Related
It belongs to LLM concepts in my stack, under Generative AI and LLMs. Others in the same group: