SAINT

Overview

SAINT stands for Self-Attention and Intersample Attention Transformer. It is a deep learning model for tabular data that adds a second kind of attention to the usual transformer recipe. Normal self-attention lets the columns of one row talk to each other. Intersample attention lets the rows in a batch talk to each other. The idea is that similar rows can help predict each other, much as a doctor compares a new patient with earlier patients who look alike.

SAINT also adds a self-supervised pre-training stage. It corrupts rows with data-mixing tricks and trains the network to match the corrupted version to the original through a contrastive loss and a denoising loss. This helps most when labels are scarce but plenty of unlabeled rows exist. Because it sees both columns and rows as things that exchange information, SAINT is an important step toward the table-reading models described at /model/architecture and /model/research.

What it is

SAINT is a neural architecture and a training recipe for supervised learning on tables. It has four main pieces.

• An embedding layer that maps every column of a row to a vector of the same size.
• A stack of hybrid blocks, each with a self-attention step across features and an intersample-attention step across rows.
• A prediction head that reads the first token of each row and outputs a class or a number.
• An optional self-supervised pre-training stage using CutMix, mixup, contrastive learning and denoising.

It works on mixed data. Numeric columns and categorical columns are both embedded into the same space, so the rest of the network treats them alike.

What problem it tries to solve

Gradient boosted trees such as XGBoost and LightGBM were hard to beat on tables for years. Neural networks had three weaknesses on this kind of data.

1. They struggled with mixed numeric and categorical inputs.
2. They did not use the relationships between rows, even though nearest-neighbor ideas show that rows carry information about each other.
3. They had no good pre-training story, so unlabeled data was wasted.

SAINT answers each of these: unified embeddings, intersample attention, and contrastive pre-training.

What it is not

SAINT is not a foundation model that can be used on any table without training. A SAINT model is trained for one dataset schema. Its ideas, though, appear again in in-context learners such as TabPFN and TabICL, which also let rows influence each other.

Where it came from

SAINT was introduced by Somepalli and coauthors in 2021 in the paper titled "SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training". It followed a wave of attention-based tabular work.

• TabNet (2019), see TabNet, brought sequential attention for feature selection.
• TabTransformer (2020) applied transformers to categorical embeddings.
• AutoInt (2019) used self-attention over embedded features.
• FT-Transformer (2021), see FT-Transformer, appeared at about the same time and gave a clean feature-token baseline.

SAINT's distinctive contribution was to look across rows. Contrastive learning had already changed image and text learning, as in SimCLR-style methods, and SAINT brought a table-friendly version of it. The data-mixing tool CutMix, first used for images, was adapted to tabular rows, along with mixup in embedding space.

Why the timing mattered

By 2021 the community had many tabular datasets with few labels, such as medical and industrial records. Self-supervised learning was a fast-rising topic. SAINT showed that the same playbook, pre-train on unlabeled data and then fine-tune, could be adapted to tables. It also showed that a model could use the batch itself as a kind of memory.

Later influence

• Row-wise attention reappears in many later tabular architectures.
• The corruption-and-reconstruct idea reappears in Masked cell modeling and in denoising approaches, see Denoising autoencoders.
• Cross-table work such as XTab uses transformer bodies of this family.
• In-context tabular models take the row-to-row idea to its end point, where a whole labeled table becomes the context.

How it works

Picture a classroom. Each row is a student, and each column is a subject score. Self-attention across columns is like a student noticing that their math and physics scores move together. Intersample attention is like a student looking at classmates with similar profiles to guess how they themselves might do on a new test. SAINT does both, layer after layer.

Step 1: Embed every feature

Each feature of a row is mapped to a vector of the same length.

• Categorical features use an embedding table.
• Numeric features pass through a small learned network, a per-column linear layer followed by a nonlinearity in the paper's setup.

A row with d features becomes d vectors. A special CLS token is added in front, as in language models.

Step 2: Self-attention across features

Within one row, every feature vector looks at all the others and mixes in what it finds. This is the standard multi-head attention. After this step each feature vector contains information about how it relates to the other features in the same row.

Step 3: Intersample attention across rows

This is the new part. Take a batch of n rows, each with d feature vectors of size k. For each row, SAINT flattens its d vectors into one long vector of size d times k. It then treats the n long vectors as a sequence and applies self-attention across them. Each row now reads from other rows in the batch, weighted by similarity. After the step, the long vector is reshaped back to d vectors.

A simple analogy: after a meeting among the columns inside each row, the rows hold a second meeting where each row asks "who in this batch looks like me, and what do they know?" The answers are blended in.

Step 4: Feed-forward and repeat

Each attention step is followed by a feed-forward network, residual connections and layer normalization. A SAINT block runs the feature attention first and the row attention second. Several blocks are stacked, though SAINT is shallow in practice, with only a few layers because tables are small.

Step 5: Predict

The final CLS vector for each row goes into a small multilayer network. The output is a class probability or a regression value. Loss is cross-entropy or mean squared error.

Step 6: Optional contrastive pre-training

For pre-training, SAINT creates a corrupted version of every row in two ways.

• CutMix in the input space. For each row, a random subset of its raw feature values is replaced by values from another random row in the batch. The row keeps most of its identity but is partially swapped.
• Mixup in the embedding space. After embedding, the corrupted row's vectors are blended with those of another row using a mixing weight.

The original row and its corrupted version are both passed through the network. Two losses are used.

1. Contrastive loss. The representation of the clean row and of its own corrupted copy should be close, while representations of different rows should be far apart. This is related to the loss in Contrastive learning and InfoNCE.
2. Denoising loss. A small head tries to reconstruct the original feature values from the corrupted row's representation. See Denoising autoencoders.

After pre-training, the model is fine-tuned with labels using the supervised objective. If labels are plentiful, pre-training adds less. If labels are scarce, it adds more.

How intersample attention behaves at inference

At test time, a test row is placed in a batch together with training rows or other test rows. The row attention step lets the test row draw on its neighbors. In practice, the paper's setup often forms batches from the data at hand, and the behavior depends on batch composition. This is a subtle point: predictions can change slightly with batch makeup, so batching at inference needs care. A stable approach is to pair each test batch with a fixed set of training rows.

Cost

Feature attention costs grow with the square of the feature count. Row attention costs grow with the square of the batch size, but because rows are flattened into one vector, the row step has a large vector size. For tables with many columns the flattened vector gets large, which is why SAINT has a variant that reduces embedding size when the feature count is high.

Key concepts and terms

A worked example

This example sketches intersample attention in PyTorch to make the reshaping step clear. It is a simplified teaching version. See PyTorch for the framework.

import torch
import torch.nn as nn

class IntersampleAttention(nn.Module):
    """Rows in a batch attend to each other."""
    def __init__(self, n_features, d, heads=4):
        super().__init__()
        self.flat = n_features * d
        self.attn = nn.MultiheadAttention(self.flat, heads, batch_first=True)
        self.norm = nn.LayerNorm(self.flat)

    def forward(self, x):                      # x: (batch, n_features, d)
        b, f, d = x.shape
        rows = x.reshape(1, b, f * d)          # treat the batch as one sequence
        out, _ = self.attn(rows, rows, rows)
        rows = self.norm(rows + out)
        return rows.reshape(b, f, d)

The key line is x.reshape(1, b, f * d). It turns a batch of rows into a single sequence of length b, where each element is a flattened row. Normal attention across features would instead treat each row as its own sequence of length f.

Feature attention block

class FeatureAttention(nn.Module):
    def __init__(self, d, heads=4):
        super().__init__()
        self.attn = nn.MultiheadAttention(d, heads, batch_first=True)
        self.norm = nn.LayerNorm(d)

    def forward(self, x):                      # (batch, n_features, d)
        out, _ = self.attn(x, x, x)
        return self.norm(x + out)

Stacking them

class SaintBlock(nn.Module):
    def __init__(self, n_features, d):
        super().__init__()
        self.feat = FeatureAttention(d)
        self.row = IntersampleAttention(n_features, d)
        self.ff = nn.Sequential(nn.Linear(d, 2 * d), nn.GELU(), nn.Linear(2 * d, d))
        self.norm = nn.LayerNorm(d)

    def forward(self, x):
        x = self.feat(x)
        x = self.row(x)
        return self.norm(x + self.ff(x))

CutMix on a batch of raw rows

def cutmix_rows(x, p=0.3):
    """x: (batch, n_features). Swap a random subset of columns with another row."""
    b, f = x.shape
    perm = torch.randperm(b)
    mask = (torch.rand(b, f) < p)
    return torch.where(mask, x[perm], x)

Numeric walk-through

Imagine a batch of 4 rows with 3 features and an embedding size of 8. After embedding, the tensor has shape 4 by 3 by 8. Feature attention keeps this shape, with each row mixing its 3 vectors. Intersample attention flattens each row to a vector of 24 numbers, forms a 4 by 4 attention table over rows, mixes the four 24-number vectors, and reshapes back to 4 by 3 by 8. If row 2 is very similar to row 3, the attention weights between them will be high, and each row's new representation will contain part of the other's. A final CLS vector is read for each row and sent to the head.

For pre-training on this batch, apply cutmix_rows to the raw values, embed both the clean and corrupted batches, compute the contrastive loss between matching rows, and reconstruct the raw values from the corrupted representation with the denoising head.

Variants and ablations

The paper studied versions of the model that use only one kind of attention, so you can see what each part adds.

A useful reading of these ablations is that row attention helps most when the table has many rows with real similarity structure, and helps little when rows are independent noise around a global rule.

The pre-training loss in code

import torch.nn.functional as F

def contrastive_loss(z_clean, z_noisy, temperature=0.7):
    z_clean = F.normalize(z_clean, dim=-1)
    z_noisy = F.normalize(z_noisy, dim=-1)
    logits = z_clean @ z_noisy.T / temperature      # (batch, batch)
    labels = torch.arange(z_clean.size(0))
    return F.cross_entropy(logits, labels)

def pretrain_step(model, proj, recon_head, x_raw, opt, lam=10.0):
    x_noisy = cutmix_rows(x_raw)
    h_clean = model.encode(x_raw)
    h_noisy = model.encode(x_noisy)
    l_con = contrastive_loss(proj(h_clean), proj(h_noisy))
    l_den = F.mse_loss(recon_head(h_noisy), x_raw)
    loss = l_con + lam * l_den
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

In plain words, the diagonal of the similarity table is the "correct match": row i should be most similar to its own corrupted copy. The denoising term keeps the representation rich enough to rebuild the original values. The weight on the denoising term is a tunable setting.

What the model learns from row attention

Row attention can learn several behaviors, depending on the data.

• Soft nearest neighbor. Rows with similar features get high weights and share information.
• Group effects. Rows from the same hidden group, such as the same store or machine, pool what they know.
• Outlier isolation. A row that matches nothing in the batch attends broadly and gets a flat blend, which can mark it as unusual.
• Label smoothing in disguise. Neighbors with similar targets pull a noisy row toward the local average.

Data regimes where it shines

1. Few labels, many rows. Pre-training uses the unlabeled rows.
2. Clustered rows. For example, repeated measurements of the same entity.
3. Moderate feature count. Row attention stays affordable.
4. Noisy features. Corruption training makes the model robust to damaged inputs.

Data regimes where it struggles

1. Very wide tables. The flattened row vector is large.
2. Independent rows with a simple rule. Trees do just as well at lower cost.
3. Strict online scoring. A single record has no batch to attend to.
4. Extremely small tables. There are too few rows to make attention useful.

Comparing batch size choices

Serving a SAINT model

A practical serving design keeps a small, fixed reference set of training rows in memory. Each incoming record is placed with that reference set, scored, and only its own output is returned. This keeps scores stable and makes a single-record request possible. The reference set should be refreshed on a schedule, and its selection should cover the main clusters in the data so that every query has useful neighbors.

Strengths and limitations

Strengths

• Uses row relationships. Similar rows help each other, which is useful when the data has clusters or near-duplicates.
• Handles mixed types. All columns share one embedding space.
• Benefits from unlabeled data. Contrastive pre-training helps in low-label settings.
• Competitive among neural models. In the original paper it outperformed several earlier tabular networks and was competitive with boosting on many tasks.
• Flexible tasks. It works for classification and regression.
• Natural fit for imputation and denoising. The corruption objective trains it to repair damaged rows.

Limitations

• Batch dependence. Predictions can depend on which other rows are in the batch.
• Memory and compute. Row attention over large batches with flattened rows can be heavy, especially for many columns.
• Tuning effort. Embedding size, depth, dropout, mixing probability and learning rate all need care.
• Not a universal model. It is trained per dataset and cannot read an unseen schema.
• Trees often remain strong. On many datasets a tuned tree ensemble matches or beats it at much lower cost.
• Benefit of pre-training is uneven. It helps with few labels but adds little with plentiful labels.
• Harder to explain. Predictions that depend on neighbors are harder to audit than a single tree path.

How it compares with alternatives

SAINT sits between classic neural models and in-context learners. It has the row-to-row idea of nearest neighbors and the feature-to-feature idea of transformers, but it learns both through gradient training on one dataset.

Role in tabular foundation models

SAINT is not a foundation model, but it contains two ideas that foundation models for tables rely on: reading rows and columns as two axes of attention, and pre-training without labels.

Row and column structure

A table is a grid, and information flows along both axes. In SAINT, feature attention moves information along a row and intersample attention moves it along a column of rows. Modern table models make this explicit with alternating column-wise and row-wise attention, closely related to Axial attention from vision. If you want a model that reads any table, you need both axes, because the meaning of a cell depends on its row and on how other rows look in that column. See /model/architecture.

Mixed types

SAINT embeds numbers and categories into one space. Foundation models need the same step, with the added difficulty that the embedding must work for columns never seen in training. A common solution is to generate column-specific embeddings from the column's values, such as quantiles or sample statistics, rather than using fixed per-column weights.

Missing values

SAINT does not natively model missing cells, but its corruption objective is close. If CutMix replaces cells and the network learns to recover the originals, the same machinery can learn to fill in a blank. A dedicated missing token or mask makes the link explicit. This leads directly to Masked cell modeling and Imputation networks.

No column names

SAINT is name-blind: its embeddings are tied to column positions and learned per column. A model that must handle tables with unnamed or shuffled columns needs symmetry, where permuting columns does not change the output. The attention in SAINT respects this for the feature step, but the flattening in the row step does depend on column order, because the concatenated vector has a fixed layout. This is a point where later designs improve things, for instance by using row attention per column instead of per flattened row.

Pre-training on synthetic or real tables

SAINT pre-trains on the same dataset it later classifies, so there is no transfer across tables. Foundation models push the idea much further. Cross-table work such as XTab pre-trains over many real datasets. TabPFN pre-trains on a huge supply of synthetic tables drawn from random causal models. The corruption objectives from SAINT remain useful as a source of training signal for either. For benchmark comparisons see /model/benchmarks.

In-context learning

The row-to-row attention in SAINT is the ancestor of in-context learning for tables. If the rows in the batch carry labels, and the model is trained so that a query row reads the labeled rows to predict its own label, you get an in-context learner. SAINT does not do this by default, since labels are not part of the input during row attention, but the machinery is nearly the same. Models like TabPFN and TabICL make labels part of the context and train on many tables so the procedure generalizes.

Continual learning

When new data arrives, a SAINT model can be fine-tuned, but it may forget older patterns. Techniques such as Experience replay, Elastic weight consolidation and Synaptic intelligence can limit forgetting. Row attention also offers a gentle path, since adding new rows to the reference set changes behavior without changing weights.

Imputation and anomaly detection

The denoising head predicts original values from corrupted input, which is a form of imputation. The reconstruction error is also a signal for Anomaly detection: rows that cannot be reconstructed well are unusual. Row attention adds a neighborhood view, so a row can be judged against its look-alikes rather than against global averages.

Sector and semantic classification

When labeling rows or tables by domain, such as telling a retail table from an industrial one, representations that reflect both column interactions and row neighborhoods help. A classification head over the CLS vector is a basic version of what Semantic sector heads do at larger scale, where text features from column names add semantic signal that SAINT alone does not use.

Practical guidance and best practices

When SAINT is a good choice

• You have a medium to large table with mixed types.
• You have many unlabeled rows and few labels.
• You expect near-duplicate or clustered rows that could help each other.
• You want a neural model to complement trees in an ensemble.

When to avoid it

• The table is tiny. A simple model or an in-context learner is safer.
• You need strict row independence at inference, such as scoring one record at a time without a reference set.
• Latency and cost limits rule out transformers.

Data preparation

1. Split first and fit preprocessing on training rows only.
2. Standardize numeric columns, or apply a quantile transform for skewed ones.
3. Map categories to integer codes and reserve a code for unknown values.
4. Remove ID columns and columns that leak the label.
5. Decide how missing values will be handled and apply the same rule at serving.

Training recipe

1. Optionally pre-train with CutMix, mixup, contrastive and denoising losses on all available rows, labeled or not.
2. Fine-tune with the supervised loss on labeled rows.
3. Use early stopping on a validation split.
4. Repeat with several seeds.

Hyperparameters

Use a tuner such as Optuna and compare against trees given the same budget.

Inference with row attention

Decide on a policy and keep it fixed.

• Score test rows in batches that include a fixed anchor set of training rows.
• Or disable the row step at inference if you need row independence, accepting some loss in quality.
• Always test whether predictions change when the batch order or composition changes.

Explaining results

Use permutation importance or SHAP on the final predictions. For neighbor-based reasoning, inspect the row attention weights to see which training rows influenced a prediction, but treat this as a clue and not proof.

Evaluating whether pre-training helped

Run a simple ablation before adopting the extra stage.

1. Train without pre-training on 5, 10, 25 and 100 percent of the labels.
2. Train with pre-training on the same label fractions.
3. Plot validation score against label fraction for both.
4. Keep pre-training only if the gap is clear at the label level you will really have.

If the curves meet at your real label count, skip pre-training and save the compute.

Monitoring in production

• Track the input distribution of each column against training statistics.
• Track the share of unseen categories.
• Track the average maximum row-attention weight. A rise can mean queries no longer have close neighbors.
• Re-check calibration on fresh labeled data on a regular schedule.

Common mistakes

• Skipping the fair baseline. Compare with a tuned tree ensemble, not a default one.
• Leaking via pre-training. If the test rows are used in unlabeled pre-training, the test set is no longer clean unless this matches the deployment setting.
• Ignoring batch effects. Different batch compositions at inference can shift scores.
• Too much corruption. A very high CutMix probability destroys the signal and makes the contrastive task trivial or impossible.
• Too many columns with a large embedding. The flattened row vector becomes huge and memory runs out.
• Unscaled numerics. Large raw values destabilize the embedding layer.
• Using pre-training on plentiful labels and expecting big gains. The benefit is mainly for low-label settings.
• Forgetting seeds. Neural results vary, so average several runs.
• Treating attention as explanation. Attention weights show information flow, not cause and effect.

Frequently asked questions

What does SAINT stand for?

SAINT stands for Self-Attention and Intersample Attention Transformer. The name describes its two kinds of attention, one that mixes the features inside a row and one that mixes information between different rows in a batch.

What is intersample attention?

Intersample attention is a step where each row in a batch is flattened into a single vector and the rows attend to each other. A row can then borrow information from rows that look similar, which acts like a learned version of nearest-neighbor lookup inside the network.

How is SAINT different from FT-Transformer?

FT-Transformer uses one token per feature and attention across features only. SAINT adds attention across rows and a contrastive pre-training stage. FT-Transformer is simpler and easier to reproduce, while SAINT can use similarities between rows and unlabeled data.

Does SAINT need pre-training?

No. It can be trained directly with a supervised loss. Pre-training with CutMix, mixup, contrastive and denoising objectives is optional, and it tends to help most when labeled rows are few and unlabeled rows are plentiful.

What is CutMix for tables?

CutMix for tables replaces a random subset of a row's feature values with the values from another random row. The corrupted row keeps most of its information, so the network can learn to match it to the original row and to recover the swapped values.

Is SAINT better than gradient boosting?

Not reliably. In the original study it beat several neural baselines and was competitive with boosted trees on many datasets, but trees remain strong and much cheaper. Many teams use SAINT as one member of an ensemble rather than as a replacement.

Can SAINT handle missing values?

Not directly in the basic design, so values should be imputed or flagged beforehand. The denoising objective is related to imputation, and a dedicated missing token could be added, which is the approach taken in masked-cell methods.

Why can predictions depend on the batch?

Because row attention lets each row read from the other rows in the same batch. If the batch contains different rows, the attention weights change, so the representation and the output can shift slightly. A fixed reference set at inference keeps behavior stable.

Is SAINT a tabular foundation model?

No. It is trained on one dataset at a time and cannot read a new schema without training. It does, however, share key ideas with foundation models, namely attention over rows and columns and pre-training without labels.

How does it relate to in-context learning?

Row attention is the closest ancestor. If labeled rows are included in the context and the model is trained across many tables to predict a query row from them, you get an in-context learner such as TabPFN. SAINT trains on a single table and does not use labels in the context by default.

Glossary

• Batch: a group of rows processed together.
• CLS token: a learned token whose final vector summarizes the row.
• Contrastive learning: learning by pulling matched pairs together and pushing different ones apart. See Contrastive learning.
• CutMix: mixing two examples by swapping a portion of one into the other.
• Denoising: recovering the original input from a corrupted one.
• Embedding: a learned vector standing for a feature value.
• Fine-tuning: continued training on labeled data after pre-training.
• Flattening: joining several vectors into one long vector.
• InfoNCE: a common contrastive loss. See InfoNCE.
• Intersample attention: attention between rows of a batch.
• Mixup: blending two examples with a weight.
• Projection head: a small network used for the contrastive objective.
• Self-attention: attention in which tokens read from tokens of the same sequence.
• Self-supervised learning: learning from labels made out of the data itself. See Self-supervised learning.
• Temperature: a scale in the contrastive loss that controls sharpness.

Summary

SAINT is a tabular transformer that applies attention along both axes of a table. Feature attention lets the columns of a row interact, and intersample attention lets the rows of a batch influence each other. Optional contrastive and denoising pre-training with CutMix and mixup lets it learn from unlabeled rows.

It was a notable 2021 step in neural tabular learning and remains a useful model when rows are clustered and labels are scarce. It is costly, sensitive to batching, and tied to one dataset, so tuned trees such as XGBoost and CatBoost still make a strong reference. Its lasting influence is conceptual. Row and column attention, corruption-based self-supervision and row-to-row reading all show up in tabular foundation models, from cross-table pre-training with XTab to in-context learners like TabPFN. For more background, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Related models and baselines in my stack, under Foundation models and applied ML. Others in the same group: