Transfer learning

Overview

Transfer learning means starting from a model that has already learned from a large source task and adapting it to a smaller target task. Instead of learning everything from random weights, the new model begins with useful knowledge. This usually means fewer labels, less compute and better accuracy on the target.

It is the standard way to build practical models today. A vision backbone trained on millions of images is adapted to spot defects on a factory line. A language model is adapted to a legal or medical domain. A tabular foundation model, pre-trained on many tables, is adapted to a new company dataset. The knowledge moves from where data is plentiful to where it is scarce. It pairs naturally with self-supervised learning, which supplies the pre-trained models.

What it is

Transfer learning is a family of techniques, not one algorithm. The shared idea is that what a model learned in one setting helps in another.

Source and target

• Source task or domain: where the model was first trained, usually with abundant data.
• Target task or domain: where you want to use it, usually with little data.

An everyday analogy

A person who plays the piano finds it easier to learn the organ. They already know notes, rhythm and finger control. They only have to learn what is new. A person who has never touched a keyboard must learn everything. Transfer learning gives a model the piano player's head start.

Why it works

Early layers of a network tend to learn general patterns, such as edges in images, common word parts in text, or basic relationships between numeric columns. Later layers tend to become specific to the source task. Reusing the general layers saves the effort of rediscovering them.

Main ways to transfer

Types by setting

• Inductive transfer: the target task differs from the source, and target labels are available.
• Transductive transfer: the task is the same but the domain differs, and labels may be missing in the target.
• Unsupervised transfer: neither side has labels for the target; used with clustering or representation reuse.

Where it came from

Reusing knowledge across tasks has been studied in machine learning for decades, often under names such as multi-task learning and inductive bias transfer. The idea became mainstream in practice with deep vision networks.

Vision

In the early 2010s researchers noticed that features learned by a network trained on a large image classification set worked well for other vision tasks. A common recipe appeared: take a pre-trained network, remove the last layer, and train a new one. Later this extended to fine-tuning all layers.

Language

For a while, language models reused only word embeddings. Around 2018, whole pre-trained language models were fine-tuned on small labelled sets and set records across many benchmarks. This showed that transfer could work for deep, contextual representations, not only for shallow ones.

Efficiency era

As models grew to billions of parameters, full fine-tuning became costly. Parameter-efficient methods such as adapters, prefix tuning, LoRA and QLoRA appeared. They train a tiny fraction of the weights while keeping the base model frozen. The wider toolkit is collected under PEFT.

Tables

Tabular transfer was slower to mature. Different tables have different columns, types and meanings, so there is no shared vocabulary like words or pixels. Progress came from models that treat a table as a set of typed cells with learned column roles, and from pre-training on very many tables, including synthetic ones. See TabPFN, TabICL and XTab.

How it works

Here is the usual workflow from choosing a source model to deploying the adapted one.

Step 1: Choose a source model

Pick a model whose pre-training data and objective resemble your target. A model trained on natural images may transfer well to product photos and poorly to medical scans. A model trained on many tables transfers best to tables with similar kinds of columns.

Step 2: Decide how much to freeze

Freezing means the weights do not change during training. The less target data you have, the more you should freeze.

• Tiny data: freeze everything and train a head.
• Small data: unfreeze the top layers.
• Larger data: unfreeze all with a small learning rate.

Step 3: Replace or add the task head

The pre-trained model ends in a head built for the source task. Replace it with one matching your target: a different number of classes, a regression output, or a ranking score.

Step 4: Set the learning rates

Use a smaller learning rate for pre-trained layers than for the new head. A large rate can wipe out what the model already knows. A common pattern is layer-wise rates that grow from bottom to top.

Analogy: renovating a house. You repaint rooms freely (the head), but you move load-bearing walls (early layers) with great care.

Step 5: Train and monitor

Train on target data with early stopping. Track validation performance and compare it with a baseline trained from scratch.

Step 6: Check for forgetting and drift

Fine-tuning can erase source knowledge. This is called catastrophic forgetting. If you still need the source skills, mix in some source data, use experience replay, add a penalty such as EWC, or use adapters that leave the base untouched.

Step 7: Deploy and refresh

Version the base model and the adapter together. When new target data arrives, update the adapter or the head, not the whole system, if you can.

Parameter-efficient methods in a bit more detail

Adapters

Small bottleneck layers are inserted into the network. Only these are trained. The base stays fixed.

LoRA

Instead of updating a big weight matrix, LoRA learns a small low-rank change that is added to it. It trains few parameters and the update can be merged into the base for serving. See LoRA, DoRA and QLoRA.

Prompt and prefix tuning

Learned vectors are added to the input or to attention layers. The model body is frozen. See prompt tuning and prefix tuning.

BitFit and scaling vectors

Only bias terms, or tiny scale vectors, are trained. This is extremely light. See BitFit and IA3.

What gets transferred

• Weights: the learned parameters.
• Features: outputs of a model, used as inputs to another.
• Architecture knowledge: choosing a design known to work.
• Training recipe: schedules and augmentations that worked on the source.
• Data: sometimes labelled data from a related domain is added to the target set.

Measuring transfer

Compare three numbers on the target test set:

1. A model trained from scratch.
2. A model using the pre-trained weights.
3. A strong classical baseline.

If number two is not clearly better than number one, the transfer did not help. If number three beats both, a pre-trained deep model may not be the right tool for that dataset.

Key concepts and terms

A worked example

The following example shows transfer learning in two forms: first by reusing a pre-trained encoder with a small head, then by adding a LoRA-style low-rank update in plain PyTorch.

Feature extraction with a frozen encoder

import torch
import torch.nn as nn

torch.manual_seed(0)

# Pretend this encoder was pre-trained on a large source task.
encoder = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 32))
# (load pre-trained weights here in real use)

for p in encoder.parameters():
    p.requires_grad = False          # freeze

head = nn.Linear(32, 2)              # new task head
opt = torch.optim.Adam(head.parameters(), lr=1e-2)

# small target set: 120 labelled rows
X = torch.randn(120, 10)
y = (X[:, 0] - X[:, 1] > 0).long()

for epoch in range(200):
    logits = head(encoder(X))
    loss = nn.functional.cross_entropy(logits, y)
    opt.zero_grad(); loss.backward(); opt.step()

acc = (head(encoder(X)).argmax(1) == y).float().mean()
print("train accuracy:", float(acc))

Only the head, 66 parameters, is trained. That is why this works with only 120 rows.

A low-rank update (LoRA idea) by hand

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank=4, alpha=8):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.randn(rank, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))
        self.scale = alpha / rank

    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

layer = LoRALinear(nn.Linear(32, 32))
trainable = sum(p.numel() for p in layer.parameters() if p.requires_grad)
total = sum(p.numel() for p in layer.parameters())
print(trainable, "trainable of", total)

The base matrix has 1,056 parameters (including bias) and stays fixed. The two small matrices add 256 trainable ones. Because B starts at zero, the layer behaves exactly like the original at the start of training, so adaptation begins from the pre-trained behavior. Scaling this idea to a billion-parameter model is what makes cheap adaptation possible.

Reading the result

• If the frozen-encoder model already reaches good accuracy, the source features were useful for the target.
• If the LoRA-style version beats the frozen one, the target needs a small change inside the network.
• If neither beats a simple logistic regression on raw features, the source and target may be too different, or the task may be too easy to need a deep model.

Fine-tuning recipes in more detail

Gradual unfreezing

Train the head first with everything else frozen. Then unfreeze the top block and train a few epochs. Then unfreeze the next block, and so on. This protects the earliest layers, which hold the most general features.

Layer-wise learning rates

param_groups = [
    {"params": model.embed.parameters(),   "lr": 1e-5},
    {"params": model.block1.parameters(),  "lr": 2e-5},
    {"params": model.block2.parameters(),  "lr": 5e-5},
    {"params": model.head.parameters(),    "lr": 1e-3},
]
opt = torch.optim.AdamW(param_groups, weight_decay=0.01)

Lower layers move slowly, upper layers move faster, and the new head moves fastest.

Regularizing toward the source

Add a small penalty for moving away from the pre-trained weights. This keeps the model close to its starting point and limits forgetting. It is a simple cousin of EWC.

Mixing source data

Include a small share of source-like examples in each batch. This keeps old skills alive and gives the model a reminder of its original distribution.

Early stopping and checkpoints

Save a checkpoint each epoch. Pick the one with the best validation score. Fine-tuning often peaks early and then overfits.

Transfer across data types

The key question

Ask whether the source model learned something that the target also needs. If yes, transfer. If the only shared element is the file format, do not expect much help.

When transfer is not the answer

Sometimes the honest result is that a pre-trained model does not help. Typical signs are listed below.

• The target domain is highly specialized and no relevant source exists.
• The target set is large and unique, so training from scratch matches the pre-trained result.
• The task is simple, and a linear model or a tree ensemble already reaches the ceiling set by label noise.
• Latency or memory limits rule out the size of the available base model.
• Legal limits prevent using the available weights.

Treat a negative result as useful information, not as a failure of the method.

In these cases, a smaller purpose-built model can be better. Distilling a large model into a small one is another option, because the small model keeps much of the accuracy at a fraction of the cost.

A pre-flight checklist

1. Define the target metric and the minimum acceptable score.
2. Train a classical baseline and a from-scratch model.
3. Pick two or three candidate source models.
4. Run feature extraction with each. This is cheap and tells you which source fits best.
5. Try partial fine-tuning with the best one.
6. Compare learning curves at 10, 25, 50 and 100 percent of target labels.
7. Check results on a second split or a time-based split.
8. Test robustness: add noise, remove columns, shift the data.
9. Record the model version, preprocessing and seeds.
10. Plan how you will refresh the adapter when data changes.

Strengths and limitations

Strengths

• Data efficiency: strong results with few target labels.
• Lower compute: no need to train a large model from scratch.
• Faster development: a good starting point shortens experiments.
• Better generalization: general features reduce overfitting on small sets.
• Reuse across teams: one base model supports many tasks through small adapters.
• Works across modalities: text, images, audio and tables.

Limitations

• Negative transfer: if source and target differ too much, the source can hurt.
• Catastrophic forgetting: full fine-tuning can erase earlier skills.
• Inherited bias and errors: flaws in the source model move to the target.
• Licensing and provenance: the source data and weights may carry usage limits.
• Mismatch of inputs: a model expecting images cannot take tables without redesign.
• Size and cost of serving: large base models still need serious hardware.
• Hard to predict: whether transfer will help is often found only by trying.

How to spot negative transfer

Train a small from-scratch model on the target and compare. If the transferred model is worse on validation, or if it needs very careful tuning just to match the scratch model, treat that as a warning. Typical causes are very different domains, a pre-training task unrelated to the target, or a learning rate that was too high during fine-tuning.

How it compares with alternatives

Transfer learning is the umbrella. Fine-tuning, domain adaptation, few-shot learning and in-context learning are specific ways to carry it out.

Role in tabular foundation models

Tabular foundation models are the table version of the pre-train-then-adapt recipe. They are trained on very many tables so that a new table needs little extra work.

Row and column structure

A transferable tabular model cannot depend on a fixed set of columns. It treats each table as rows of typed cells and learns relations along both axes. What transfers is the skill of reading structure: how columns interact, how rows resemble each other, and how a target depends on features. See transformers and axial attention.

Mixed types

Source tables and target tables will contain numbers, categories, dates and text in different mixes. A transferable model uses shared encodings per type, such as a numeric embedding, a categorical embedding and a text embedding. The weights for these encoders are reused across tables, even though the columns differ.

Missing values

Pre-training on many tables with varied missingness teaches the model to cope with gaps. When the model is applied to a new table with its own missing pattern, it already knows how to use partial rows. Imputation becomes a transferred skill. See masked cell modeling and imputation networks.

No column names

When headers are absent or meaningless, a transferred model must infer roles from values and relations. Because it has seen thousands of tables, it can recognize a date-like column or an amount-like column by shape. Where headers exist, they can be added as text tokens to help.

Pre-training on synthetic or real tables

• Synthetic: generated from random causal structures, which gives unlimited variety and exact ground truth. This is the route of TabPFN and TabICL.
• Real: collected from public repositories and other sources, giving real correlations and messy details. Cross-table pre-training as in XTab uses this path.
• Mixed: combines breadth and realism.

The quality of transfer depends on how well the pre-training distribution covers the target.

In-context learning as zero-update transfer

The strongest form of transfer for tables is in-context learning. The pre-trained model reads labelled training rows and unlabelled test rows in one pass and predicts, with no weight updates. This is transfer without fine-tuning: all adaptation happens inside the forward pass. It is fast and avoids forgetting by design, but it is limited by the context window.

Fine-tuning when in-context is not enough

For large tables, or when the target differs strongly from pre-training, light fine-tuning can help. Options include training only a head, adding LoRA-style modules, or updating a small part of the model. This extends the reach of the foundation model to bigger datasets.

Continual learning

Over time, new tables and new domains arrive. Sequentially fine-tuning would cause forgetting. Tabular foundation models use the toolkit of continual learning: replay of earlier examples, weight regularization such as EWC, and synaptic intelligence, together with small adapters per domain. This keeps old skills while adding new ones.

Anomaly detection

A transferred model that knows what normal tables look like can score how surprising a row is on a new table, even with few normal examples. This is useful in domains where anomalies are rare and unlabelled. See anomaly detection.

Sector and semantic classification

Features learned across many tables can be reused to classify what a column or dataset represents, such as its business sector or semantic type. A small head trained on a modest labelled set can use the transferred representation. See semantic sector heads.

For wider context on how these pieces are combined, see the research, architecture and benchmarks pages.

Practical guidance and best practices

• Always run a from-scratch or classical baseline. It tells you whether transfer helped.
• Match source and target as closely as possible in data type, scale and domain.
• Start light: feature extraction first, then partial fine-tuning, then full fine-tuning only if needed.
• Use a smaller learning rate for pre-trained layers. A common starting point is ten times smaller than for the head.
• Use warm-up and early stopping to protect pre-trained weights.
• Preprocess target data the same way as the source. Mismatched scaling or encoding is a frequent cause of failure.
• For tables, keep numeric scaling and category handling consistent with the pre-trained model's expectations.
• Prefer parameter-efficient methods for large models. They save memory and make rollback easy.
• Store adapters separately from the base so you can switch tasks without copying the whole model.
• Use cross-validation on small targets, since a single split is noisy.
• Check for leakage between source and target. If the target appears in the pre-training data, scores will be inflated.
• Respect licences and data terms attached to the pre-trained model.
• Document what was transferred, from where, and how, for later audits.

Choosing a strategy

Common mistakes

• Skipping the baseline, so there is no proof that transfer helped.
• Using a learning rate that is too high, which erases useful pre-trained features.
• Unfreezing everything on a tiny dataset and overfitting.
• Forgetting to apply the same preprocessing as in pre-training.
• Choosing a source model from a very different domain and expecting a miracle.
• Ignoring forgetting when the source skills still matter.
• Leaking target data into the pre-training set or the validation set.
• Comparing a tuned transferred model against an untuned scratch model.
• Treating the pre-trained model as a black box with no checks on bias or errors.
• Not versioning the base model, so results cannot be reproduced after an update.
• Using full fine-tuning for each new task and storing a full copy every time, when adapters would do.

Frequently asked questions

What is transfer learning in simple words?

It is the practice of taking a model that already learned something useful on a large task and adapting it to a new, smaller task. The model keeps its general knowledge and only learns what is different. This saves data, time and compute.

What is the difference between transfer learning and fine-tuning?

Transfer learning is the broad idea of reusing knowledge from one task in another. Fine-tuning is one way to do it, by continuing to train the pre-trained weights on the new task. Other ways include feature extraction, adapters and in-context learning.

When does transfer learning fail?

It fails when the source and target are too different, a case known as negative transfer. It can also fail when fine-tuning uses a learning rate that is too high and destroys useful features, or when preprocessing differs from what the model expects. A scratch baseline helps detect these problems.

Can I use transfer learning on tabular data?

Yes. Tabular foundation models are pre-trained on many tables, often synthetic ones, so that they can handle a new table with little or no training. You can use them in context, train a small head on their features, or add light adapters for larger datasets.

What is catastrophic forgetting?

It is the loss of earlier knowledge when a model is trained on new data. It happens because the weights that served the old task are overwritten. Replay, regularization such as EWC and the use of adapters are common remedies.

How is transfer learning related to self-supervised learning?

Self-supervised learning is a common way to create the pre-trained model, by learning from unlabelled data. Transfer learning is the step of reusing that model for a downstream task. They are two stages of the same pipeline.

Should I freeze all layers?

Freeze more when you have very little target data and the target is close to the source. Unfreeze more when you have more data or the target differs. Many practitioners start with a frozen model and a new head, then unfreeze the top layers if the score plateaus.

What are parameter-efficient methods?

They are techniques that adapt a large model by training only a small number of added or selected parameters. Examples are adapters, LoRA, prefix tuning and bias-only tuning. They reduce memory and storage and make it easy to keep one base model with many small task-specific add-ons.

Is in-context learning a kind of transfer learning?

Yes. In-context learning adapts a pre-trained model to a new task by putting examples in its input, without changing weights. For tabular models, labelled rows act as the prompt. It is a form of transfer that needs no training on the target.

How much target data do I need?

It depends on the closeness of the source and target and on the method. Feature extraction can work with tens or hundreds of labelled examples. Fine-tuning usually needs more, and adapter methods sit in between. Always test with a learning curve to see how performance changes with the amount of data.

Glossary

• Adapter: a small trainable module added to a frozen model.
• Backbone: the main pre-trained feature network.
• Catastrophic forgetting: loss of old knowledge when learning new things.
• Domain shift: a difference in data distribution between source and target.
• Feature extraction: using a frozen model's outputs as input features.
• Fine-tuning: further training of pre-trained weights on a target task.
• Freezing: stopping weights from changing during training.
• Head: the final layers that produce a task output.
• In-context learning: adapting through examples in the input with no weight updates.
• Inductive transfer: transfer when the target task differs from the source task.
• Low-rank update: a compact weight change used by LoRA.
• Negative transfer: harm caused by reusing a mismatched source.
• Pre-trained model: a model already trained on a large source task.
• Source domain: where the model was originally trained.
• Target domain: where the model will be used.
• Warm-up: a short period of gradually rising learning rate at the start of training.

Summary

Transfer learning reuses what a model learned on a large source task so that a small target task needs fewer labels and less compute. Ways to do it range from freezing the model and training a head, to fine-tuning, to parameter-efficient adapters, to in-context use with no weight updates. The main risks are negative transfer, forgetting and mismatched preprocessing.

For tabular foundation models, transfer is the central idea: pre-train on many synthetic or real tables so the model understands row and column structure, mixed types and missing values, then apply it to a new table with in-context learning or light tuning. Continual-learning tools protect earlier skills, and the same representations support imputation, anomaly detection and semantic classification. Always compare against a baseline and match preprocessing. Related reading: self-supervised learning, contrastive learning and transformers.

Related

It belongs to Learning paradigms in my stack, under AI and machine learning. Others in the same group: