Özer Cem KELAHMETKELAHMET
Tools · Related models and baselines

FT-Transformer

FT-Transformer turns every table feature into a token and runs a transformer over them. Learn how it works, where it wins, and how it fits tabular foundation models.

Overview

FT-Transformer, short for Feature Tokenizer plus Transformer, is a deep learning model for tabular data. It converts every column of a row into a vector, called a token, and then lets a standard transformer mix those tokens with self-attention. A special summary token is read at the end to make the prediction. The model works for classification and regression, and it treats numeric and categorical columns in one uniform way.

It matters for two reasons. First, it gave the field a strong, simple, well-tested neural baseline that is often competitive with gradient boosted trees on many datasets. Second, its design, one token per column and a shared transformer body, became a building block for later work. Cross-table pre-training, self-supervised tabular learning and modern tabular foundation models all reuse the idea that a table row is a short sequence of feature tokens. See also Transformers for the general architecture and /model/architecture for how tabular models are built from these parts.

What it is

FT-Transformer is an architecture, not a library or a service. It is a recipe with two parts.

  • A feature tokenizer that maps each input feature to a vector of the same size, for example 192 numbers.
  • A transformer encoder that is a stack of ordinary attention and feed-forward blocks, applied to the sequence of those vectors.

A table row with 20 columns becomes a sequence of 20 tokens, plus one extra learned token at the front. The extra token is often called the CLS token, borrowed from language models. After the transformer layers, the CLS token's final vector is passed through a small head to produce the output.

What makes it "tabular"

Text models read words in a fixed order. A table row has no natural order. The columns can be shuffled and the meaning does not change. FT-Transformer respects this because each feature gets its own learned embedding identity, and attention does not depend on position. There are no positional encodings in the model. The identity of the column is carried by the tokenizer's own parameters.

Typical input and output

  • Input: a row with numeric columns and categorical columns. Missing values must be handled before or inside the pipeline.
  • Output: a class score vector for classification, or a single number for regression.
  • Size: usually small by modern standards, from a few hundred thousand to a few million parameters, because tables are small compared to text.

Where it came from

The model was introduced in the paper "Revisiting Deep Learning Models for Tabular Data" by Gorishniy and coauthors, presented at NeurIPS in 2021. The paper had a practical goal. By 2021 there were many neural tabular models, each tested on different datasets with different tuning budgets, so it was unclear which ones really worked.

The authors did two things.

  1. They built a fair benchmark with a shared set of datasets and a shared tuning protocol.
  2. They showed that two simple models were strong: a ResNet-like multilayer network adapted to tables, and the new FT-Transformer.

The conclusion was balanced. Gradient boosted decision trees were still very strong, and no neural model won everywhere. But FT-Transformer was the best deep model across many tasks and came close to or beat boosting on several. It was also the more universal of the two baselines.

  • TabTransformer (2020) applied a transformer only to categorical features and passed numeric features around it. FT-Transformer extends the idea so that numeric features are also tokenized.
  • AutoInt (2019) used self-attention over embedded features for click prediction. It is a close ancestor.
  • TabNet (TabNet) used sequential attention for feature selection in 2019.
  • SAINT (SAINT) added attention across rows and contrastive pre-training in 2021.

What came after

A follow-up study in 2022 looked at how to embed numeric features. It showed that richer numeric embeddings, such as periodic functions or piecewise linear encodings, can lift many tabular networks, including FT-Transformer. The model was also used as the backbone in cross-table pre-training work such as XTab. Researchers also use it as a baseline when they present newer tabular foundation models like TabPFN, TabICL and TabSTAR.

How it works

Think of a row as a short meeting. Each column sends one delegate. Every delegate first writes a short personal note about their value. Then the delegates take turns listening to each other and updating their notes. At the end, a chairperson, the CLS token, summarizes the meeting into a decision. Below is the step by step process.

Step 1: Feature tokenization

Each feature j gets its own tokenizer.

  • For a numeric feature with value x, the token is x * W_j + b_j. Here W_j is a learned vector for that column and b_j is a learned bias vector. In plain words, the number scales a column-specific direction in embedding space.
  • For a categorical feature, the token is a lookup in an embedding table for that column, plus a bias vector. Each category has its own learned vector.

Everyday analogy: imagine every column owns a set of colored stamps. A numeric value picks how strongly to press one stamp. A category picks which stamp to use. The result is always a card of the same size, so cards from different columns can be placed on the same table.

Step 2: Add the CLS token

A learned vector is prepended to the sequence. It starts with no information about the row. Its job is to collect information through attention.

Step 3: Transformer layers

The sequence passes through several identical blocks. Each block has:

  • Multi-head self-attention. Every token compares itself with every other token, decides how much to listen to each, and mixes in their information. Different heads can focus on different kinds of relationships, such as "age and income" or "city and product".
  • A feed-forward network applied to each token on its own. The paper used a gated activation called ReGLU.
  • Residual connections and normalization. The model uses pre-normalization, where layer normalization is applied before each sub-block. A detail from the paper is that the first block skips the first normalization, because the tokens are already well scaled.

Step 4: Read the CLS token

After the last block, only the CLS token is kept. It goes through layer normalization, a ReLU, and a linear layer. For regression the result is a number. For classification it is class logits.

Step 5: Train

Training uses standard supervised loss: cross-entropy for classification, mean squared error for regression. The optimizer is typically AdamW with weight decay and a tuned learning rate, and early stopping on a validation set. Dropout in attention and feed-forward layers is a key knob.

Why it can find feature interactions

In a tree, interactions are found by stacking splits. In an FT-Transformer, interactions come from attention. If the model needs to know that "high balance combined with low age" matters, the tokens for those two columns can attend to each other in an early layer, and later layers can build on the result. Each additional layer allows higher-order combinations.

Cost

Self-attention compares all pairs of tokens, so cost grows with the square of the number of columns. For 20 or 50 columns this is cheap. For thousands of columns it becomes heavy, and the authors noted this as a limit. Later work reduces cost with approximations or by grouping columns. Related ideas such as Axial attention and Perceiver IO address scaling in other ways.

Key concepts and terms

Term Plain meaning
Feature tokenizer The part that turns each column value into a fixed-size vector
Token One vector that stands for one feature of one row
CLS token A learned extra token that gathers a summary of the row
Self-attention A step where every token chooses which other tokens to read from
Multi-head attention Several attention patterns computed in parallel
Embedding dimension The length of each token vector
Residual connection A shortcut that adds the input of a block to its output
Pre-normalization Normalizing before a sub-block rather than after
ReGLU A gated feed-forward activation used in the original model
Numeric embedding A learned way to turn a number into a vector
Attention map The table of weights showing which tokens read which
Early stopping Halting training when validation loss stops improving

A worked example

This example shows a small FT-Transformer in PyTorch. It is simplified for clarity and not tuned. See PyTorch for the framework.

python
import torch
import torch.nn as nn

class NumericTokenizer(nn.Module):
    def __init__(self, n_features, d):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(n_features, d) * 0.02)
        self.bias = nn.Parameter(torch.zeros(n_features, d))

    def forward(self, x):                      # x: (batch, n_features)
        return x.unsqueeze(-1) * self.weight + self.bias

class CategoricalTokenizer(nn.Module):
    def __init__(self, cardinalities, d):
        super().__init__()
        self.embs = nn.ModuleList([nn.Embedding(c, d) for c in cardinalities])
        self.bias = nn.Parameter(torch.zeros(len(cardinalities), d))

    def forward(self, x):                      # x: (batch, n_cat) of ints
        out = [emb(x[:, i]) for i, emb in enumerate(self.embs)]
        return torch.stack(out, dim=1) + self.bias

class SmallFTT(nn.Module):
    def __init__(self, n_num, cardinalities, d=64, layers=3, heads=4, out=1):
        super().__init__()
        self.num_tok = NumericTokenizer(n_num, d)
        self.cat_tok = CategoricalTokenizer(cardinalities, d)
        self.cls = nn.Parameter(torch.zeros(1, 1, d))
        block = nn.TransformerEncoderLayer(
            d_model=d, nhead=heads, dim_feedforward=2 * d,
            dropout=0.1, batch_first=True, norm_first=True)
        self.encoder = nn.TransformerEncoder(block, num_layers=layers)
        self.head = nn.Sequential(nn.LayerNorm(d), nn.ReLU(), nn.Linear(d, out))

    def forward(self, x_num, x_cat):
        tokens = torch.cat([self.num_tok(x_num), self.cat_tok(x_cat)], dim=1)
        cls = self.cls.expand(tokens.size(0), -1, -1)
        h = self.encoder(torch.cat([cls, tokens], dim=1))
        return self.head(h[:, 0])

Walk-through with numbers

Suppose a row has 3 numeric columns (age, income, tenure) and 2 categorical columns (region, plan type). With embedding size 64 the tokenizer produces 5 vectors of length 64. Adding the CLS token gives a sequence of 6 tokens. Each attention layer builds a 6 by 6 table of weights per head. The CLS row of that table tells how strongly the summary reads from each feature. After three layers, the CLS vector holds a mixed description of the whole row, and the head turns it into a probability of, for example, churn.

Training loop sketch

python
model = SmallFTT(n_num=3, cardinalities=[12, 4], d=64)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=1e-5)
loss_fn = nn.BCEWithLogitsLoss()

for epoch in range(100):
    model.train()
    for xb_num, xb_cat, yb in train_loader:
        opt.zero_grad()
        loss = loss_fn(model(xb_num, xb_cat).squeeze(-1), yb)
        loss.backward()
        opt.step()
    # evaluate on validation and apply early stopping here

Preprocessing that matters

Numeric columns should be scaled, for example with standardization or a quantile transform. Heavy-tailed columns often benefit from a quantile transform to a normal shape. Categorical values must be mapped to integer codes, with a reserved code for unseen or missing categories.

Numeric embeddings in more depth

The simple linear tokenizer can only draw a straight line through embedding space as a number grows. Real columns are rarely that simple, so later work proposed richer options.

  • Linear: x * W + b. Cheapest and the original choice.
  • Periodic: the number is passed through sine and cosine functions with learned frequencies, then a small linear layer. This lets the model express repeating or sharply changing patterns.
  • Piecewise linear encoding: the number line is cut into bins, and the value is described by how far it fills each bin. This works like a soft histogram.
  • Target-aware bins: bin edges are chosen with help from a small decision tree, so the bins match useful cut points.

The practical lesson is that the tokenizer is not a minor detail. On columns with thresholds, such as "age over 65", a better numeric embedding can matter as much as depth or width.

Model variants and design choices

Choice Option A Option B Effect
Summary readout CLS token Mean over tokens CLS is the original; mean pooling is simpler and sometimes similar
Normalization Pre-norm Post-norm Pre-norm trains more stably on small data
Activation ReGLU GELU Gated units were used in the paper
Column dropout None Randomly drop feature tokens Acts as regularization and prepares for missing data
Numeric embedding Linear Periodic or binned Richer embeddings help on irregular columns
Depth 1 to 2 layers 4 to 6 layers Shallow is often enough on small tables

Why a single token per feature is a strong prior

A neural network that sees a row as a flat vector must learn which positions are related. By giving each feature its own token, FT-Transformer builds in the idea that features are separate objects that talk to each other. This is like seating each person at a round table instead of squeezing them into one chair. The model can then learn who should talk to whom, and it can reuse the same attention machinery for every pair.

Training behavior to expect

  • Training loss often falls fast and validation loss turns around early, so early stopping with patience is important.
  • Small learning rates work better than for typical image models.
  • Results vary between seeds, so reporting the mean and spread over several runs is good practice.
  • Larger batches speed up training, but very large ones can hurt generalization on small data.
  • Learning-rate warmup and a decay schedule help, though the original setup used simple settings.

Using it as a feature extractor

After training, the CLS vector is a compact description of the row. You can feed it to a tree model, use it for nearest-neighbor search, or cluster it. Because the representation was trained for a supervised task, it captures the parts of the row that matter for that target. Training with a self-supervised objective first gives a representation that is less tied to one label.

Handling very wide tables

If a table has thousands of columns, there are several workable approaches.

  1. Group related columns, build one token per group, and attend over groups.
  2. Use a low-rank or linear attention variant to avoid the full pairwise table.
  3. Select features first with a tree model, then train the transformer on the top ones.
  4. Use random column subsets per step, as in column dropout, and average at inference.

A decision checklist

  • Is the table small, under a few thousand rows? Start with trees or an in-context model such as TabPFN.
  • Is the table large with many interacting features? A tuned FT-Transformer is worth trying.
  • Do you need one model across many tables? Look at cross-table pre-training such as XTab or text-aware models such as TabSTAR.
  • Do you need fast inference on CPU? Prefer trees.
  • Do you need a simple, auditable model? Prefer trees or linear models with SHAP checks.

Strengths and limitations

Strengths

  • Uniform treatment of types. Numbers and categories become the same kind of object, so one model handles both.
  • Strong deep baseline. In the original study it was the best neural model on average and competitive with boosting on many datasets.
  • Simple and standard. It uses off-the-shelf transformer blocks, so tooling, mixed precision and hardware support work out of the box.
  • Useful interpretability signal. Attention maps give a rough view of which features interact, though they should not be treated as proof of importance.
  • Good backbone for pre-training. Because each column is a token, the model can be pre-trained with masking or contrastive objectives and reused.
  • Flexible heads. The same body can drive classification, regression or multi-task outputs.

Limitations

  • Cost grows with the square of column count. Very wide tables are expensive.
  • Needs tuning. Learning rate, dropout, embedding size and depth all matter. Out-of-the-box defaults can lag behind tuned trees.
  • Slower than trees. Training and inference are heavier than for XGBoost, LightGBM or CatBoost on CPU.
  • Not always better. On small, noisy or highly irregular tables, boosted trees often win. Trees handle sharp thresholds and uninformative columns very well.
  • Column identity is fixed. The tokenizer has parameters tied to each column, so a trained model cannot be applied to a table with different columns without retraining.
  • Missing values need handling. The base design has no built-in missing indicator, so imputation or an extra flag is needed.
  • Sensitive to numeric scale and outliers unless the preprocessing is careful.

How it compares with alternatives

Method Core idea Handles raw mixed types Needs training per dataset Typical strength
FT-Transformer One token per feature plus transformer Yes, with preprocessing Yes Strong deep baseline, good backbone
XGBoost / LightGBM / CatBoost Ensembles of decision trees Yes (CatBoost handles categories natively) Yes, but fast Hard to beat on many tables
Random forest Many independent trees Yes Yes Robust, little tuning
TabNet Sequential attention picks features Yes Yes Built-in feature selection
SAINT Attention across features and rows Yes Yes Uses similar rows, pre-training
XTab Shared transformer pre-trained on many tables Yes, via per-table featurizers Fine-tune Transfer between datasets
TabPFN Pre-trained network doing in-context prediction Yes No gradient training, one forward pass Small and medium tables
TabICL In-context learning scaled to larger tables Yes No gradient training Larger context sets

A common guideline is to start with a tree ensemble as the reference, add a tuned FT-Transformer as a diverse second model, and blend the two. Because their errors differ, ensembles of trees and neural models often beat either alone.

Role in tabular foundation models

FT-Transformer is not itself a foundation model. It is trained from scratch on one table at a time. Yet it is one of the clearest stepping stones to models that learn from raw tables in general, because it fixes the basic shape of the problem: a row is a set of typed cells, and each cell becomes a token.

Row and column structure

A tabular foundation model must respect that rows are exchangeable examples and columns are unordered features. FT-Transformer shows how: no positional encoding, per-column identity in the tokenizer, and attention over columns. Foundation models extend this by adding attention across rows, as in SAINT and in in-context learners such as TabPFN, where the labeled rows are the "prompt". Details of such architectures are on /model/architecture.

Mixed types

The tokenizer is a clean answer to mixed data. Numbers use a linear or periodic embedding, categories use lookup embeddings. Newer models keep this separation, but make the tokenizer shared across tables, which is the hard part.

Missing values

The plain model has no special treatment for missing cells. In foundation models this becomes a design choice. One approach is a learned "missing" embedding in place of the cell's token. Another is to mask the token and let attention fill the gap. FT-Transformer's token layout makes both easy, and it is the reason masked-cell objectives are natural. See Masked cell modeling and Imputation networks.

No column names

FT-Transformer ignores column names. Each column is known only by its index and its learned parameters. A model that must work on any unseen table cannot depend on fixed per-column parameters, so foundation models replace them with parameters generated from the data, such as column statistics, or from a language model reading the name. TabSTAR goes the second way and uses text encoders to make tokens from names and values, which links tabular and text understanding. When names are missing, models must infer a column's role from its values alone.

Pre-training on synthetic or real tables

The standard FT-Transformer is trained from random initialization. XTab used it as the shared body and pre-trained it across many datasets, with a separate input featurizer and output head for each table. In contrast, TabPFN pre-trains on synthetic tables generated from random causal structures, so that at test time no new training is needed. Both approaches rely on the same idea of a transformer over cell tokens. Results of such comparisons are discussed at /model/benchmarks and the research background is at /model/research.

In-context learning

FT-Transformer learns by gradient descent on the target table. In-context learners instead read labeled rows at inference and predict in one pass. The two can be seen as two ends of a spectrum: the first stores knowledge in weights tuned per table, the second stores a general learning procedure in weights and applies it to new data at run time.

Continual learning

A per-table FT-Transformer that is trained again on new data risks forgetting. Foundation models that keep learning need tools such as Elastic weight consolidation or Experience replay. The transformer body of FT-Transformer is compatible with these methods since they act on weights and training data, not on the architecture.

Imputation and anomaly detection

Because each cell is a token, a masked-token objective trains the model to predict a hidden cell from the rest of its row, which is imputation. The same reconstruction error can serve as an anomaly score: rows that the model reconstructs poorly are unusual. See Anomaly detection and Isolation forest for classical alternatives.

Semantic and sector classification

When columns carry meaning, such as names that point to a business domain, a model that only knows column indices wastes that signal. Row-level tokens from FT-Transformer can feed a classification head that labels a table or row by its domain, and text-aware variants add the semantics from names. See Semantic sector heads for how a head on top of table representations can assign sector labels.

Practical guidance and best practices

When to use it

  • You have enough rows, often thousands or more, and moderate column count.
  • You want a neural model to blend with trees, or you want embeddings of rows for downstream use.
  • You plan to pre-train with masking or contrastive objectives on a large pool of related tables.

Data preparation

  1. Split the data first, then fit preprocessors on training data only.
  2. Scale numeric columns. Try a quantile transform for skewed columns.
  3. Encode categories as integers. Reserve an index for unknown and missing categories.
  4. Add a missing-indicator flag for important numeric columns, or impute carefully.
  5. Drop identifier-like columns with unique values, which invite overfitting.

Hyperparameters to tune

Parameter Typical range Note
Embedding size 64 to 512 Larger needs more regularization
Layers 1 to 6 Few layers are often enough
Heads 4 to 8 Must divide embedding size
Attention dropout 0 to 0.5 Important for small data
Feed-forward dropout 0 to 0.5 Tune with attention dropout
Learning rate 1e-5 to 1e-3 Use AdamW
Weight decay 1e-6 to 1e-3 Often not applied to biases and norms

Use a search library such as Optuna with a fixed budget, and compare to a similarly tuned tree model. Reporting both under the same budget is the only fair way to judge. See Bayesian optimization for the principle behind many tuners.

Evaluation

  • Use cross-validation or several random splits. Single splits are noisy on small tables.
  • Choose a metric that fits the task, such as AUC for ranking, log loss for probabilities, RMSE for regression.
  • Check calibration when probabilities matter.

Explaining predictions

Attention maps are a useful hint but are not feature importance. Use permutation importance or SHAP for stronger statements, or LIME for local explanations.

Operational tips

  • Train with mixed precision on a GPU for speed on larger tables.
  • Track runs with MLflow or a similar tool so results are reproducible.
  • Export the preprocessing together with the model; mismatched scaling is a common production bug.

A fair comparison recipe

Many disputed results about tabular deep learning come from unequal effort. A fair test follows a short recipe.

  1. Pick a set of datasets with different sizes, column counts and type mixes.
  2. Fix the splits and the metric before any tuning.
  3. Give every model the same tuning budget, for example 100 trials each.
  4. Tune on a validation set, never on the test set.
  5. Repeat the final training with several seeds and report the mean and spread.
  6. Report training time and inference time next to accuracy.

Below is a short sketch of such a loop with a search library.

python
import optuna

def objective(trial):
    cfg = {
        "d": trial.suggest_categorical("d", [64, 128, 256]),
        "layers": trial.suggest_int("layers", 1, 4),
        "dropout": trial.suggest_float("dropout", 0.0, 0.4),
        "lr": trial.suggest_float("lr", 1e-5, 1e-3, log=True),
    }
    return train_and_validate(cfg)   # returns validation loss

study = optuna.create_study(direction="minimize")
study.optimize(objective, n_trials=100)

Debugging checklist

When the model does badly, work through the list in order.

  • Does a trivial baseline, such as predicting the mean or majority class, beat the model? If so, the pipeline has a bug.
  • Is the training loss falling? If not, lower the learning rate and check the scaling of inputs.
  • Is the validation loss far above the training loss? Raise dropout or weight decay, or cut depth.
  • Do predictions look constant? Check for dead activations or a target that was not scaled in regression.
  • Are categories mapped consistently between training and serving?
  • Does shuffling the column order change results? It should not. If it does, something is using position by mistake.
  • Do identical rows get identical outputs in evaluation mode? If not, dropout is still on.

Reproducibility notes

  • Set seeds for the framework, the data loader and the splitter.
  • Record library versions and the preprocessing pipeline.
  • Keep the category-to-index map as a saved artifact.
  • Log the full configuration for each run, so that a result can be rebuilt later.

Small experiment ideas to build intuition

  • Train with one layer and with four layers, and compare validation loss on the same split.
  • Remove the CLS token and use mean pooling, and see whether anything changes.
  • Shuffle the column order at test time and confirm that predictions stay the same.
  • Replace the linear numeric tokenizer with a binned one on a column that has a threshold effect.
  • Mask 10 percent of cells at random during training and test how the model copes with missing data at inference.

These small tests show quickly how much of the performance comes from architecture and how much from preprocessing.

When the target is skewed

For regression with a long-tailed target, such as prices or counts, train on a transformed target, for example a logarithm, and invert the transform at prediction time. Evaluate on the original scale too, because a good score on the log scale can hide large errors on big values.

Multi-task and auxiliary heads

One body can feed several heads, for example one for the main label and one that reconstructs masked cells. The auxiliary head acts as a regularizer and often helps on small data.

Common mistakes

  • Comparing against untuned baselines. A tuned tree model is the right reference. A default one makes a neural model look better than it is.
  • Skipping numeric scaling. Raw values with large ranges destabilize the linear tokenizer.
  • Leaking information through preprocessing. Fitting scalers or encoders on the full dataset leaks test information.
  • Treating identifiers as features. Row ids and timestamps used naively can let the model memorize.
  • Over-deep models on small data. More layers do not help when rows are few; reduce depth and raise dropout.
  • Reading attention as causality. Attention shows where information flows in the network, not which feature causes the outcome.
  • Ignoring class imbalance. Use weighting or a suitable metric when one class is rare.
  • Using one random seed. Neural training varies between seeds. Average several runs.
  • Applying the model to a table with different columns. The tokenizer is tied to the original columns, so it needs retraining or a cross-table approach.

Frequently asked questions

What does FT-Transformer stand for?

It stands for Feature Tokenizer plus Transformer. The feature tokenizer turns each column value of a row into a vector, and the transformer then mixes those vectors with self-attention so the model can learn how features interact before it makes a prediction.

Is FT-Transformer better than XGBoost?

Not in general. In the original study it was the strongest neural model and matched or beat boosting on some datasets, but gradient boosted trees remained very strong overall. The best choice depends on the table, and many practitioners train both and combine them in an ensemble.

Does FT-Transformer need a GPU?

A GPU is not required for small tables, but it makes training much faster, especially with many rows or large embeddings. On a modest dataset with a small model, a CPU run can finish in a reasonable time, though tree models will still be faster.

How does FT-Transformer handle categorical features?

Each categorical column has its own embedding table. The category code selects a learned vector from that table, and a column-specific bias is added. The result is a token of the same size as a numeric token, so both types enter the transformer together.

Can FT-Transformer handle missing values?

The basic architecture does not include a built-in missing-value mechanism, so you should impute values or add missing-indicator features before training. Some extensions add a learned embedding for missing cells or mask the cell token, which is closer to what foundation models do.

How is FT-Transformer different from TabNet and SAINT?

TabNet selects features step by step with sequential attention and sparse masks. SAINT adds attention across rows and a contrastive pre-training stage. FT-Transformer keeps to the simpler idea of one token per feature with a standard transformer, which makes it easy to reproduce and extend.

Can it be pre-trained?

Yes. Because each cell is a token, you can mask cells and predict them, or use contrastive objectives. Cross-table work such as XTab pre-trains a shared FT-Transformer-style body over many datasets with table-specific input and output parts, then fine-tunes on a target table.

Why does it have no positional encoding?

Columns in a table have no natural order, so position carries no meaning. Column identity is instead stored in the parameters of the feature tokenizer, which gives each column its own weights, and attention then works on the set of tokens without caring about their order.

Is FT-Transformer a tabular foundation model?

No. It is trained on one dataset at a time and cannot be applied to a new table without training. It is, however, a close ancestor of foundation models, since it established tokenized cells and transformer layers as a standard design for learning from tables.

How many columns can it handle?

Dozens to a few hundred columns are fine. Because attention compares every pair of tokens, cost grows with the square of the column count, so very wide tables need approximations, column grouping or a different architecture.

Glossary

  • Attention head: one of several parallel attention computations inside a layer.
  • AdamW: a gradient optimizer with decoupled weight decay.
  • Backbone: the shared body of a network that feeds task-specific heads.
  • Cardinality: the number of distinct values in a categorical column.
  • CLS token: a learned summary token placed at the start of the sequence.
  • Contrastive learning: training that pulls similar examples together and pushes different ones apart. See Contrastive learning.
  • Embedding: a learned vector standing for a value, category or feature.
  • Feature tokenizer: the module that converts column values to tokens.
  • Gradient boosted trees: models that add small decision trees one at a time to fix earlier errors.
  • Head: the final layers that turn a representation into an output.
  • Layer normalization: a technique that rescales activations in each layer for stable training.
  • Logit: the raw score before a probability function.
  • Pre-normalization: applying normalization before a sub-block.
  • Quantile transform: a mapping of values to a chosen distribution based on their ranks.
  • Residual connection: a shortcut that adds a block's input to its output.
  • Self-supervised learning: learning from signals made from the data itself. See Self-supervised learning.
  • Token: a vector that stands for one piece of the input.
  • Transfer learning: reusing knowledge learned on one task for another. See Transfer learning.

Summary

FT-Transformer converts each feature of a table row into a token, adds a summary token, and applies a standard transformer so that features can learn how to interact. It was introduced in 2021 as a strong and simple deep baseline for tabular data, and it showed that neural models can be competitive with gradient boosted trees on many tasks, though not on all.

Its main value today is as a design pattern. One token per cell, no positional order, per-type embeddings and a shared transformer body are now common ground for models that learn from raw tables. Cross-table pre-training with XTab, row attention in SAINT, and in-context learners like TabPFN and TabICL all build on or react to it. For practical work, compare it against tuned trees, scale inputs carefully, and treat attention as a hint rather than an explanation. For the wider picture, see /model/research, /model/architecture and /model/benchmarks.

It belongs to Related models and baselines in my stack, under Foundation models and applied ML. Others in the same group:

Type to search.
    ↑↓ navigate↵ open