Transformers

Overview

Transformers are neural networks built around self-attention. Self-attention lets every element of an input look at every other element and decide how much each one matters for the current computation. The input can be words, image patches, audio frames or table cells. Because the same mechanism works for all of them, the transformer has become the default backbone of modern deep learning.

The architecture was introduced in 2017 for machine translation and quickly replaced recurrent networks in language tasks. Since then it has spread to vision, speech, proteins, time series and tables. For tabular data it matters because a table is a set of cells with relationships between rows and between columns. Attention can model those relationships directly. This is why many tabular foundation models, such as TabPFN, TabICL and TabSTAR, are transformers at heart.

What it is

A transformer is a stack of identical blocks. Each block has two main parts. The first part is a self-attention layer. The second part is a small feed-forward network applied to each position on its own. Around both parts there are residual connections and normalization layers that keep training stable.

The model does not read its input one step at a time. It looks at all positions in parallel. That is a large change from older sequence models such as the LSTM or GRU, which pass a hidden state along a chain.

Think of a meeting room. In a recurrent network, people whisper a message down a line, and the last person only hears a faded version. In a transformer, everyone sits around one table and can speak to anyone directly. Information does not have to travel through a long chain.

The three common shapes

• Encoder only: reads the whole input and produces a rich representation for each position. Used for classification, retrieval and tabular prediction.
• Decoder only: predicts the next element given the previous ones, using a causal mask so a position cannot see the future. Used for text generation.
• Encoder and decoder: the encoder reads the source, the decoder writes the target while attending to the encoder output. Used for translation and summarization.

What a transformer is not

• It is not tied to language. Any data that can be cut into pieces (tokens) can be processed.
• It is not magic about order. It has no built-in sense of sequence, so order must be injected through positional information, or deliberately left out when the data is an unordered set.
• It is not a single model. It is a design pattern with many variants.

Where it came from

Before transformers, sequence tasks relied on recurrent networks. Those networks were hard to parallelize and struggled with long distances between related words. Researchers added attention on top of recurrent networks around 2014 and 2015. Attention let a decoder look back at all encoder states instead of one compressed vector.

In 2017 the paper "Attention Is All You Need" removed the recurrence entirely and kept only attention. That design trained much faster on modern accelerators and gave better translation quality. It is the origin of the term transformer.

Milestones in broad strokes

• 2018: encoder-only pre-training with masked words became popular for language understanding. Decoder-only pre-training with next-word prediction scaled in parallel.
• 2020: the vision transformer showed that images cut into patches could be processed by the same architecture. See ViT.
• 2020 onward: scaling studies showed that loss falls smoothly as model size, data and compute grow together.
• 2021 onward: transformers were adapted to tabular data, for example the FT-Transformer and SAINT.
• 2022 onward: prior-fitted tabular transformers showed that a model pre-trained on synthetic tables can predict on a new table in a single forward pass. See TabPFN.

The history of the idea is also a history of removing assumptions. Convolutions assumed locality. Recurrence assumed order. The transformer assumed little and let the data decide.

How it works

This section walks through one transformer block step by step.

Step 1: Turn the input into tokens

The input is cut into pieces called tokens. For text, a tokenizer splits words into sub-words, for example with BPE or SentencePiece. For an image, tokens are square patches. For a table, a token can be a cell, a column, or a whole row.

Step 2: Embed each token

Each token is mapped to a vector of numbers called an embedding. The vector is learned. Tokens with similar roles end up with similar vectors.

Analogy: an embedding is like a name tag that carries many facts at once, such as category, size and role, written as numbers.

Step 3: Add position information

Attention alone treats the input as a bag. To tell the model where each token sits, a positional signal is added. Common choices are:

• Sinusoidal encodings: fixed waves of different frequencies.
• Learned position embeddings: one vector per position.
• Relative or rotary schemes: encode the distance between tokens inside the attention score.

For tables, order of rows usually carries no meaning, so models often drop row positions and use column identity instead.

Step 4: Compute queries, keys and values

For every token the model computes three vectors using three learned matrices:

• Query: what this token is looking for.
• Key: what this token offers.
• Value: the content this token will pass on.

Analogy: a library. You walk in with a question (query). Each book has a label on its spine (key). You compare your question with every label, then read the contents (value) of the books that match best.

Step 5: Score, scale and normalize

The score between token i and token j is the dot product of the query of i with the key of j. Scores are divided by the square root of the key size, which keeps them in a stable range. A softmax turns each row of scores into weights that sum to one.

Step 6: Mix the values

The output for token i is the weighted sum of all value vectors, using the weights from the previous step. Tokens that matter more contribute more.

The core formula is short:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d)) V

Step 7: Use several heads

One attention pattern is rarely enough. The model splits the vectors into several smaller groups, called heads, and runs attention in each one. One head may track nearby words, another may track subjects and verbs, another may track matching columns. The head outputs are joined and mixed by another learned matrix.

Step 8: Apply a feed-forward network

After attention, each token passes through a small two-layer network. Attention moves information between positions. The feed-forward network processes information within a position. Much of the model's stored knowledge lives in these layers.

Step 9: Residuals and normalization

Each sub-layer adds its input back to its output (a residual connection) and applies layer normalization. Residuals let gradients flow through many layers. Normalization keeps activations in a healthy range.

Step 10: Stack blocks and read out

Dozens of blocks are stacked. The last layer's vectors feed a task head: a softmax over a vocabulary for text, a class logit for classification, or a number for regression.

Masks

A mask is a rule saying which positions may attend to which.

• Padding mask: ignore filler positions.
• Causal mask: a position sees only earlier positions.
• Custom masks: in some tabular models, test rows can see training rows but training rows cannot see test rows.

Cost

Full attention compares every token with every other token, so cost grows with the square of the sequence length. Much engineering work targets this cost. Examples are memory-efficient kernels such as FlashAttention, factorized schemes such as axial attention, and alternatives such as Mamba.

Variants and refinements

The original design has been refined in many directions. Knowing the main families helps you read papers and pick tools.

Efficiency variants

• Sparse attention: each token attends to a chosen subset of positions, such as a local window plus a few global tokens.
• Linear attention: rewrites the softmax so cost grows linearly with length, at some cost in quality.
• Grouped or multi-query attention: several heads share keys and values, which shrinks the memory needed when generating text.
• Memory-efficient kernels: compute exact attention but avoid writing the full score matrix to slow memory.

Architecture variants

• Pre-norm blocks: normalization is applied before each sub-layer, which makes very deep stacks easier to train.
• Gated feed-forward layers: a gate decides how much of each hidden unit passes through.
• Mixture of experts: only some feed-forward experts run for each token.
• Latent bottlenecks: a small set of learned latent vectors reads from a large input, as in Perceiver IO.

Training variants

• Masked prediction: hide some tokens and recover them, as in masked autoencoders and masked cell modeling.
• Next-token prediction: predict what comes next, the base of most generative language models.
• Contrastive objectives: pull related views together, as in contrastive learning.
• Denoising: corrupt an input and train the model to restore it, related to denoising autoencoders.

Why so many variants exist

The basic block is general, but each domain has a different bottleneck. Language needs long contexts. Vision needs many patches. Tables need to treat rows as an unordered set and columns as typed fields. Most refinements change one part of the block to relieve that bottleneck and leave the rest alone.

Key concepts and terms

A worked example

The following code builds attention from scratch in plain Python with NumPy for three tokens, so the numbers can be followed by hand. It then shows a minimal PyTorch encoder for tabular rows.

Attention by hand

import numpy as np

# Three tokens, each with a 2-dimensional embedding
X = np.array([[1.0, 0.0],
              [0.0, 1.0],
              [1.0, 1.0]])

# Learned projections (set to identity to keep the example readable)
Wq = np.eye(2)
Wk = np.eye(2)
Wv = np.eye(2)

Q, K, V = X @ Wq, X @ Wk, X @ Wv
d = Q.shape[1]

scores = Q @ K.T / np.sqrt(d)

def softmax(z):
    z = z - z.max(axis=1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)

weights = softmax(scores)
out = weights @ V

print(np.round(weights, 3))
print(np.round(out, 3))

Reading the result: the third token has equal overlap with the first two, so it spreads its attention across both and itself. The first token attends mostly to itself and to the third, because their vectors overlap. The output of each token is a blend of the values, weighted by those attention weights.

A tiny encoder for table rows

import torch
import torch.nn as nn

class RowEncoder(nn.Module):
    def __init__(self, n_features, d_model=64, n_heads=4, n_layers=2, n_classes=2):
        super().__init__()
        # one small linear map per feature turns each number into a token
        self.feature_embed = nn.ModuleList(
            [nn.Linear(1, d_model) for _ in range(n_features)]
        )
        self.cls = nn.Parameter(torch.zeros(1, 1, d_model))
        layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=n_heads, batch_first=True
        )
        self.encoder = nn.TransformerEncoder(layer, num_layers=n_layers)
        self.head = nn.Linear(d_model, n_classes)

    def forward(self, x):                      # x: (batch, n_features)
        tokens = [emb(x[:, i:i+1]) for i, emb in enumerate(self.feature_embed)]
        tokens = torch.stack(tokens, dim=1)    # (batch, n_features, d_model)
        cls = self.cls.expand(x.size(0), -1, -1)
        z = self.encoder(torch.cat([cls, tokens], dim=1))
        return self.head(z[:, 0])              # read out from the CLS token

model = RowEncoder(n_features=8)
logits = model(torch.randn(16, 8))
print(logits.shape)  # torch.Size([16, 2])

What this shows:

• Each feature becomes its own token, so attention can learn which columns relate.
• A special summary token collects information from all features.
• No positional encoding is used, because column order is not meaningful here. Column identity is carried by the separate embedding for each feature.

This is the same idea used by the FT-Transformer, written in a very small form.

Where transformers are used

Why one design fits so many fields

The block only asks for a set of vectors. Once a domain has a way to produce such vectors, everything else is reused: the attention layers, the training tricks, the hardware kernels and the open-source tooling such as PyTorch and Hugging Face. This reuse is a large practical reason for the architecture's spread. A team that knows how to train one transformer can apply most of that knowledge to a new domain.

What changes between domains

The list below names the parts a practitioner must adapt.

• The tokenizer, because each data type has its own natural pieces.
• The positional scheme, because order means different things in a sentence, a picture and a table.
• The pre-training task, because the best signal differs: next word, masked patch, masked cell.
• The size of the context, because an image or a large table can produce thousands of tokens.

Strengths and limitations

Strengths

• Parallel training: all positions are processed at once, which suits GPUs and other accelerators.
• Long-range links: any two tokens connect in one step, regardless of distance.
• Flexible inputs: text, images, audio, tables and graphs fit after suitable tokenization.
• Scaling behavior: quality tends to improve steadily with more data, parameters and compute.
• Transfer: one pre-trained model can serve many tasks through fine-tuning, prompting or in-context learning.
• Interpretability aids: attention weights offer a partial view of what the model used, though they are not a full explanation.

Limitations

• Quadratic cost: memory and time grow with the square of sequence length for full attention.
• Data hunger: large transformers need large pre-training sets to beat simpler models.
• No built-in order: positions must be supplied, and extrapolating beyond the trained length can fail.
• Weak inductive bias on small data: with a few hundred rows, a tree ensemble can still win unless the model is pre-trained.
• Numeric handling: raw numbers are awkward for tokens. Models need careful number embeddings or binning.
• Opaque behavior: large models can be hard to audit, and attention maps do not prove causal use.
• Compute and energy: training and serving large models is expensive.

How it compares with alternatives

A useful rule: transformers excel when there is a lot of data or a strong pre-trained model, and when relationships between elements matter more than local patterns. Trees remain strong when data is small and there is no pre-trained model to lean on.

Role in tabular foundation models

A tabular foundation model is a model pre-trained on very many tables so that it can predict on a new table without being rebuilt from scratch. Most such models are transformers, and the reasons are specific to how tables look.

Rows and columns are two axes of structure

A table is a grid. Relationships exist along rows (one record has several fields) and along columns (a field has a distribution across records). A plain text transformer sees one long sequence and mixes these up. Tabular transformers handle the two axes explicitly.

• Attention across columns inside a row: learns that income and age together say something that neither says alone.
• Attention across rows inside a column or a batch: lets a test row compare itself with labelled training rows, much like a nearest-neighbour method but learned.
• Alternating the two: one layer attends along features, the next along rows. This is the idea behind axial attention and is used in several prior-fitted tabular models.

Mixed types

Real tables mix numbers, categories, dates and free text. Tokenization choices matter:

• Numbers can be passed through a small learned map, binned, or embedded with periodic features.
• Categories can use a learned embedding per value, or be encoded by position in the column.
• Text columns can be embedded by a language encoder and fed in as one more token. TabSTAR follows this idea.

Missing values

Attention handles gaps in a natural way. A missing cell can be given a special token or simply omitted from the set the other cells attend to. Because the model sees many tables with many missingness patterns in pre-training, it can learn to fill gaps from context. See masked cell modeling and imputation networks.

No column names

Many tables arrive with headers like col_1 or with meaningless names. A model that depends on names fails there. A transformer can instead infer a column's role from its values and from how it relates to the other columns. The attention pattern becomes a kind of learned column identity. Models can also use names when they exist, which helps when headers are informative.

Pre-training on synthetic or real tables

Two sources of pre-training data are common.

• Synthetic tables: sampled from a wide family of random causal graphs and noise models, so the model sees millions of different "datasets". This is the route taken by TabPFN and TabICL.
• Real tables: collected from open repositories and corporate-style sources. Real data brings true correlations but is limited in volume and carries licensing and privacy limits.

Mixed strategies are also used. The transformer is the learner that absorbs either source.

In-context learning

In a language model, in-context learning means the examples are written in the prompt. In a tabular transformer the labelled rows are the prompt. The model reads the training rows and the unlabelled test rows together and outputs predictions in one forward pass, with no gradient updates for that dataset. Attention is the mechanism that lets each test row look up the relevant training rows.

Continual learning

Fine-tuning a transformer on new tables can cause forgetting of older skills. Methods such as EWC and experience replay help keep earlier abilities while adding new ones. Parameter-efficient methods such as LoRA and the wider PEFT family add small trainable pieces and leave the base weights alone, which also reduces drift.

Imputation and anomaly detection

Because the transformer models the joint structure of a row, the same network can be used for several tasks.

• Imputation: mask a cell and predict it from the rest of the row and from similar rows.
• Anomaly detection: score how surprising a cell or row is under the learned structure. See anomaly detection.

Sector and semantic classification

Heads on top of the encoder can classify what a column or table is about, for example which business sector a dataset belongs to. This builds on the same representations. See semantic sector heads.

More detail on how these ideas are combined can be found on the pages for research, architecture and benchmarks.

Practical guidance and best practices

• Start with a strong baseline. Run a gradient-boosted tree model first, so you know what the transformer must beat.
• Match the tokenization to the data. For text use a sub-word tokenizer. For tables decide whether tokens are cells, columns or rows.
• Normalize numbers. Standardize or quantile-transform numeric columns before embedding.
• Use pre-trained weights when you can. Training a large transformer from scratch on a small table rarely pays off.
• Watch the context window. For in-context tabular models, large tables may need sub-sampling or chunking of the training rows.
• Prefer efficient kernels. Memory-efficient attention such as FlashAttention cuts memory and time.
• Use mixed precision. Half-precision training speeds up learning if you keep a loss scaler or use bfloat16.
• Tune learning rate and warm-up first. Transformers are sensitive to learning rate, and a short warm-up avoids early divergence.
• Regularize on small data. Dropout, weight decay and early stopping matter more than extra depth.
• Prefer adapters for domain changes. LoRA or QLoRA is cheaper than full fine-tuning.
• Evaluate on held-out data that matches deployment. Leakage between train and test rows will inflate scores.
• Log attention and activation statistics. Sudden changes often signal data problems.

A short checklist before training

1. Is the task defined, with a metric that matches the business goal?
2. Is there a tree-based baseline score?
3. Are numeric and categorical columns encoded consistently between training and inference?
4. Is the split by time or by group when rows are not independent?
5. Are random seeds and library versions recorded?

Training recipes in practice

Optimizer and schedule

Most transformers are trained with an Adam-style optimizer with decoupled weight decay. The learning rate rises linearly during a warm-up phase, then decays with a cosine or linear schedule. Gradient clipping at a fixed norm prevents rare spikes from wrecking a run.

Batch size and sequence length

Larger batches give smoother gradients but need more memory. When memory is short, gradient accumulation sums gradients over several small batches before one update. Gradient checkpointing trades extra compute for lower memory by recomputing activations in the backward pass.

Monitoring a run

• Watch training and validation loss together. A widening gap signals overfitting.
• Track gradient norm. Sudden spikes often precede divergence.
• Track the fraction of attention mass on special tokens. If almost all of it lands there, heads may be wasted.
• Save checkpoints often, and keep the best one by validation score.

Scaling guidance

For a fixed compute budget, a model that is too large for its data wastes compute, and so does a small model trained on far more data than it can absorb. Balanced scaling of parameters and data is a better guide than either alone. For tabular work, the pre-training set is often synthetic, so the amount of data is limited only by generation cost, and the question becomes how diverse the generator is.

Serving

At inference time, cache what can be reused. For text, a key-value cache avoids recomputing past tokens. For in-context tabular models, the encoded training rows can sometimes be cached while new test rows stream through. Quantization to lower precision reduces memory and latency with a small loss in quality when done carefully.

Common mistakes

• Training from scratch on tiny tables. A model with millions of parameters will overfit a few hundred rows.
• Forgetting positional information on sequences, or adding it on unordered rows where it only adds noise.
• Leaking the target through features, then trusting a high score.
• Ignoring scale of numeric features, so a column with large values dominates the embedding.
• Reading attention maps as proof of reasoning. A weight shows where information flowed, not why a decision was made.
• Using too long a context for the hardware, then silently truncating input.
• Comparing against an untuned baseline. Trees and linear models also need tuning.
• Skipping a causal mask in a generative setting, which lets the model peek at the answer.
• Applying a language tokenizer unchanged to tabular text, which splits identifiers into meaningless pieces.
• Over-tuning on a validation set until it no longer reflects new data.

Frequently asked questions

What is a transformer in simple words?

A transformer is a neural network that lets every part of its input look at every other part and decide what is important. It does this with a mechanism called self-attention. The same design works for words, image patches and table cells, which is why it is so widely used.

Why are transformers better than recurrent networks?

They train in parallel because they do not wait for a hidden state to pass along a chain. They also connect distant elements in a single step, which avoids the fading memory of older recurrent models. These two traits made it practical to train very large models on very large datasets.

What does self-attention actually compute?

For each token, self-attention builds a weighted average of all tokens in the input. The weights come from comparing a query vector of that token with the key vectors of the others, followed by a softmax. The result is a new vector for the token that mixes in the information it found most relevant.

Do transformers work on tabular data?

Yes, and they work best when pre-trained. A transformer that has seen many tables, often synthetic ones, can predict on a new table by reading labelled rows as context. Without pre-training, a transformer on a small table often loses to a well-tuned gradient-boosted tree model.

What is the difference between an encoder and a decoder?

An encoder reads the entire input at once and builds a representation for each position, so it suits understanding tasks. A decoder generates output one element at a time and only sees what came before. Some systems combine both, as in translation.

Why do transformers need positional encoding?

Self-attention treats its input as an unordered set, so on its own it cannot tell the first word from the last. Positional encoding adds a signal that marks where each token sits. For data where order has no meaning, such as table rows, models may leave it out on purpose.

What is the context window?

The context window is the maximum number of tokens a transformer can process in one pass. It exists because full attention cost grows with the square of the length. For tabular in-context models, it limits how many training rows can be shown at once.

How is a transformer different from a mixture of experts?

A mixture of experts is a way of building the feed-forward part of a transformer so that each token is sent to only a few specialist sub-networks. It keeps attention the same and lets total parameters grow without a matching rise in compute per token. It is a refinement of the transformer, not a rival to it.

Can I fine-tune a transformer cheaply?

Yes. Parameter-efficient methods such as LoRA add small trainable matrices and keep the original weights frozen, which cuts memory and storage. Quantized variants reduce memory further. These methods are often enough for domain adaptation.

Are attention weights an explanation of the model's decision?

Not fully. Attention weights show how information was mixed in a layer, which is useful for inspection. They do not capture the whole computation, because later layers and feed-forward blocks also shape the output. Treat them as a clue, not a proof.

Glossary

• Attention: a way of computing weights that say how much each element should influence another.
• Autoregressive: generating one element at a time, each conditioned on earlier ones.
• Backbone: the main network that produces features, to which task heads are attached.
• Batch: a group of examples processed together.
• Bidirectional: able to use context from both sides of a position.
• Causal mask: a mask that prevents a position from using later positions.
• Context window: the maximum input length in tokens.
• Cross-attention: attention in which queries come from one sequence and keys and values from another.
• Embedding: a learned vector that stands for a token.
• Encoder: the part that turns input into representations.
• Decoder: the part that generates output.
• Feed-forward network: a small per-position network inside each block.
• Head: one parallel attention pattern.
• In-context learning: solving a task from examples placed in the input.
• Layer normalization: rescaling of activations within a layer.
• Logit: a raw score before a softmax.
• Pre-training: learning general skills before a specific task.
• Residual connection: a shortcut adding input to output.
• Softmax: a function that turns scores into probabilities.
• Token: a unit of input such as a sub-word, a patch or a cell.

Summary

Transformers are neural networks that use self-attention to let every input element weigh every other element. They were introduced in 2017, they train in parallel, and they scale well, which is why they now underpin language, vision and many other fields.

For tables, transformers model both the column axis and the row axis, cope with mixed types and missing values, and support in-context learning when pre-trained on synthetic or real tables. They also support imputation, anomaly detection and semantic classification from one set of representations. Use them with a clear baseline, careful tokenization of numbers, and parameter-efficient adaptation when data is limited. Explore related ideas in self-supervised learning, transfer learning and contrastive learning.

Related

It belongs to Architectures in my stack, under AI and machine learning. Others in the same group: