Adapters

Overview

Adapters are small trainable modules that you insert into a large pre-trained neural network. The original weights stay frozen. Only the new modules are trained on your task. A typical adapter is a narrow two-layer network with a residual connection, so it adds a few percent of the parameters of the host model and can still reach quality close to full fine-tuning.

The main reason to use adapters is economy. You keep one copy of the big model and store a tiny file per task. Training needs less memory because no gradients or optimizer state are kept for the frozen weights. Adapters also reduce forgetting, because the original knowledge is never overwritten. They are one of the oldest and most widely used members of the PEFT family, alongside LoRA, prefix tuning, prompt tuning, IA3 and BitFit.

What it is

An adapter is a small neural sub-network placed inside a layer stack of a pre-trained model. In the classic form, it sits after a sub-layer of a transformer block. The sub-layer might be the attention block or the feed-forward block. The adapter takes the hidden vector, squeezes it into a smaller dimension, applies a non-linearity, expands it back, and adds the result to the original vector.

Think of a large factory with a fixed assembly line. You do not rebuild the line for each new product. You bolt a small fixture onto one station that adjusts the output for the new product. The line stays the same. The fixture is cheap, easy to swap, and easy to remove.

The shape of a bottleneck adapter

The standard design has four parts:

• A down-projection that maps a vector of size d to a smaller size r.
• A non-linearity such as ReLU or GELU.
• An up-projection that maps size r back to size d.
• A skip connection that adds the input back to the output.

The skip connection matters. With near-zero initial weights, the adapter starts as an identity function. The model behaves exactly like the pre-trained model at step zero, and training moves it gently away from that point.

What is trained and what is frozen

Why the word "adapter"

The term comes from the idea of adapting a general model to a specific job without rewriting it. In electronics, an adapter lets one plug fit another socket. Here the adapter lets one pre-trained backbone serve many tasks.

Where it came from

Before adapters, the usual way to reuse a pre-trained network was to fine-tune every weight, or to freeze the network and train only a new top layer. The first is expensive and creates a full model copy per task. The second is cheap but often too weak.

Adapter modules for transfer learning were introduced for text transformers around 2019 in the paper "Parameter-Efficient Transfer Learning for NLP" by Houlsby and colleagues. The idea itself had earlier roots in computer vision, where researchers proposed residual adapter modules to share one network across many image domains. The 2019 paper showed that inserting small bottleneck layers into a pre-trained BERT-style model could come close to full fine-tuning on a broad set of language tasks, while adding only a small fraction of parameters per task.

Several lines of follow-up work shaped the field:

• Pfeiffer-style adapters used a single adapter per block in a better position, which cut cost further.
• AdapterFusion proposed a second stage that learns how to combine several task adapters.
• AdapterHub and similar libraries made adapters easy to share, download and stack.
• Parallel adapters, and approaches that unify adapters with prefix methods, studied where and how to attach small modules.
• LoRA, introduced in 2021, offered a related idea that can be merged into the base weights, which removes inference overhead.

Today adapters are one tool in a larger toolkit. The PEFT library and the Hugging Face ecosystem support several of these methods through one interface. See also /model/research for the research context of reusing a single backbone across many tasks.

How it works

This section walks through the mechanism step by step.

Step 1: Start with a pre-trained backbone

You begin with a model that already learned useful features from large data. It could be a language model, a vision model, or a model trained on tables. The backbone is loaded and its weights are set to not require gradients.

Analogy: you hire an experienced generalist who already knows the field. You will not send them back to school. You give them a short briefing for your specific job.

Step 2: Insert adapter modules

At chosen places inside each layer, you add a small module. In a transformer block, the common positions are:

1. After the multi-head attention output, before the first residual add and norm.
2. After the feed-forward output, before the second residual add and norm.

Some designs use only one of these positions. Using both gives more capacity. Using one gives lower cost.

Step 3: Initialize close to identity

The down and up projections are initialized with small values, so the adapter output is almost zero. Because of the skip connection, the whole block behaves like the original. This protects the pre-trained knowledge at the start.

Step 4: Train only the new parameters

During the forward pass, data flows through the frozen model and the adapters. During the backward pass, gradients flow back through the whole network, but weight updates only apply to adapter parameters and the task head. The optimizer state is kept only for those small tensors.

This is where the memory savings come from. Optimizer states such as Adam moments are twice the size of the trainable weights. If only one or two percent of weights are trainable, that overhead nearly disappears.

Step 5: Save a small file

After training, you save only the adapter weights. The file may be a few megabytes even when the backbone has billions of parameters. To serve the task, you load the shared backbone once and attach the right adapter.

Step 6: Swap, stack or fuse

Because adapters are separate modules, you can:

• Swap them at run time, one per task or per customer.
• Stack them, for example a language adapter followed by a task adapter.
• Fuse them, by learning a small attention layer that mixes the outputs of several adapters.

The math in one line

For a hidden vector h of size d, a bottleneck adapter computes:

h_out = h + W_up( activation( W_down(h) ) )

Here W_down has shape r by d and W_up has shape d by r. The bottleneck size r is much smaller than d. The extra parameter count per adapter is about 2 times d times r, plus small biases.

Choosing the bottleneck size

The reduction factor is d divided by r. A factor of 16 means a very small adapter. A factor of 4 or 8 gives more capacity. Small tasks often work well with a high reduction factor. Harder tasks or large domain shifts benefit from a larger r.

Inference cost

Adapters add layers in series, so each forward pass has slightly more compute and some added latency. This is a real difference from LoRA, where the low-rank update can be folded into the original weights after training. If latency is critical, this extra depth is the main trade-off to weigh.

Variants

Key concepts and terms

A worked example

Below is a small, self-contained PyTorch sketch of a bottleneck adapter wrapped around a frozen linear block. It is meant to show the mechanics, not to be a production recipe.

import torch
import torch.nn as nn


class Adapter(nn.Module):
    def __init__(self, d_model: int, reduction: int = 16):
        super().__init__()
        r = max(1, d_model // reduction)
        self.down = nn.Linear(d_model, r)
        self.act = nn.GELU()
        self.up = nn.Linear(r, d_model)
        # near-identity start
        nn.init.zeros_(self.up.weight)
        nn.init.zeros_(self.up.bias)

    def forward(self, x):
        return x + self.up(self.act(self.down(x)))


class BlockWithAdapter(nn.Module):
    def __init__(self, frozen_block: nn.Module, d_model: int):
        super().__init__()
        self.block = frozen_block
        for p in self.block.parameters():
            p.requires_grad = False
        self.adapter = Adapter(d_model)

    def forward(self, x):
        return self.adapter(self.block(x))


d = 256
frozen = nn.Sequential(nn.Linear(d, d), nn.GELU(), nn.Linear(d, d))
model = BlockWithAdapter(frozen, d)

trainable = [p for p in model.parameters() if p.requires_grad]
total = sum(p.numel() for p in model.parameters())
train_count = sum(p.numel() for p in trainable)
print(f"trainable: {train_count} of {total} ({100*train_count/total:.1f}%)")

Counting parameters by hand

Take d = 256 and reduction 16, so r = 16.

• Down projection: 256 times 16 weights plus 16 biases, which is 4,112.
• Up projection: 16 times 256 weights plus 256 biases, which is 4,352.
• Adapter total: 8,464 parameters.
• Frozen block: two linear layers with about 131,584 parameters.

So the adapter is under 7 percent of this toy block. In a real model with larger width and many layers, an adapter at every layer usually lands at one to a few percent of the whole network.

Training loop outline

opt = torch.optim.AdamW(trainable, lr=1e-3, weight_decay=0.01)
loss_fn = nn.MSELoss()

for step in range(200):
    x = torch.randn(64, d)
    y = torch.randn(64, d)
    pred = model(x)
    loss = loss_fn(pred, y)
    opt.zero_grad()
    loss.backward()
    opt.step()

torch.save(model.adapter.state_dict(), "adapter_task_a.pt")

Note that only model.adapter is saved. To reuse it, you rebuild the same frozen block, create a fresh adapter, and load the saved weights.

Using a library

In practice most teams use a library. The pattern is the same across libraries: load the base model, declare a config for the adapter method, wrap the model, train, save the adapter. The PEFT library, the Hugging Face stack and trainers such as TRL follow this shape. A typical learning rate for adapters is higher than for full fine-tuning, because the new modules start from scratch.

A numeric walk-through of the data flow

Suppose a hidden vector h has width 4 and the bottleneck has width 2.

1. h = [1.0, 2.0, 0.0, -1.0]
2. After the down projection, say z = [0.5, -0.3].
3. After GELU, z becomes roughly [0.35, -0.11].
4. After the up projection, the delta might be [0.1, 0.0, -0.05, 0.2].
5. Output is h plus delta, so [1.1, 2.0, -0.05, -0.8].

The output stays close to h. That is the point: the adapter nudges the representation rather than replacing it.

Strengths and limitations

Strengths

• Very small storage per task. Many tasks share one backbone.
• Lower training memory than full fine-tuning, because optimizer states are tiny.
• Strong protection against forgetting, since base weights never change.
• Modular. You can add, remove, swap and combine adapters.
• Clean separation of tasks, which helps governance and rollback.
• Works for many architectures: text, vision, speech and tabular transformers.
• Easy to test: removing the adapter restores the original model exactly.

Limitations

• Added inference latency, because adapters are extra layers that cannot be merged away.
• Quality can trail full fine-tuning when the new task is far from pre-training data.
• More design choices than some alternatives: position, size, activation, number of adapters.
• Training can be sensitive to learning rate and initialization.
• Many separate adapters can become a management burden without good versioning.
• Gains over simpler options like LoRA are not guaranteed, so benchmark on your data.

When quality drops

If the target domain has very different statistics from pre-training, a small bottleneck may lack capacity. Signs include training loss that plateaus early and validation scores well below full fine-tuning. Raising the bottleneck size or adding adapters at more positions often helps. If it does not, continued pre-training on domain data, or full fine-tuning, may be needed.

How it compares with alternatives

How to read the table

Adapters sit in the middle on capacity. They are more expressive than prompt-based methods and bias-only methods, because they add new non-linear layers. They are less convenient than LoRA at serving time, because they cannot be merged. They are more modular than full fine-tuning.

Choosing between them

• If you need the lowest serving latency, prefer LoRA or IA3.
• If you need to compose many skills in one model, adapters and AdapterFusion are a good fit.
• If you have almost no training budget, try BitFit or prompt tuning first.
• If quality is the only goal and cost is no concern, use full fine-tuning.
• If you need to cut memory further on a large base, look at QLoRA and quantization tools such as bitsandbytes.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values, and often no useful column names. Models in this space, such as TabPFN, TabICL and TabSTAR, are pre-trained on large collections of synthetic or real tables. Adapters give a clean way to specialize such a backbone without damaging what it learned. See /model/architecture for how a table-native backbone is structured.

Adding a task without touching the backbone

A pre-trained tabular backbone encodes cells, rows and columns into vectors. A new task might be predicting churn, scoring risk, or labeling the sector of a company. With adapters, you freeze the backbone and train small modules plus a head. The same backbone then serves many tasks, each with its own adapter file.

Row and column structure

Tabular transformers often use attention across rows and across columns. Adapters can be placed in either path. A column-attention adapter can learn to weight the features that matter for a domain, such as a lab test in a health table. A row-attention adapter can adjust how the model compares one record with others. Because adapters are separate modules, you can choose which path to adapt and keep the other untouched.

Mixed types and missing values

Cells in a table may be numeric, categorical, text or missing. A pre-trained model has encoders for each type and a way to mark missing cells. When you move to a new domain, the distribution of missingness changes. A small adapter on top of the cell encoder can learn that, for example, a blank in a certain column carries meaning, without retraining the encoders for every type.

No column names

Many real tables have headers like "col_1" or abbreviations. A tabular model must then rely on values and structure. If names exist, text-aware models such as TabSTAR can use them. Adapters can help bridge between a header-free pre-training regime and a domain where names are available, by learning a light mapping from name embeddings into the feature space.

In-context learning and adapters

Some tabular foundation models predict by in-context learning: they read labeled example rows and the query rows in one forward pass, with no gradient steps. This is the approach behind TabPFN and TabICL. Adapters are a way to go beyond pure in-context use. If a domain has a stable distribution and plenty of data, a short adapter training run can move the model closer to that domain. The in-context mechanism still works afterwards, now with a shifted prior.

Continual learning

A model that must learn new domains over time faces forgetting. Adapters are a natural tool here. Each new domain gets its own adapter, and the backbone remains untouched. Compared with approaches that regularize weights, such as EWC, or replay old data, such as experience replay, adapters isolate parameters physically. The cost is that at run time you must pick or fuse the right adapter, which needs a router or a domain tag.

Imputation and anomaly detection

Imputation fills missing cells. A backbone pre-trained with masked cell modeling already predicts hidden values, and an adapter can specialize it to a table family, such as sensor logs with block-wise gaps. For anomaly detection, an adapter can learn what normal rows look like in a given system, so that reconstruction error or density scores become sharper. See also imputation networks.

Sector and semantic classification

Classifying what a table or an entity is about, for example the industry sector, needs semantic information from names and values. A small adapter plus a semantic head can map backbone features to a label set. When label sets change over time, you retrain the adapter and head, not the model.

Pre-training data matters

Pre-training on synthetic tables gives broad coverage of structure: many column types, causal graphs, noise and missingness patterns. Pre-training on real tables adds realistic semantics. In either case, the adapter step is where domain specifics enter. Evaluation on public suites is summarized at /model/benchmarks, and the choice of adapter or no adapter should always be tested against a plain baseline.

A practical pattern

1. Pick a pre-trained tabular backbone.
2. Run it zero-shot or in-context to get a baseline.
3. Add adapters and a head, and train on your labeled rows with early stopping.
4. Compare with gradient boosting such as XGBoost on the same split.
5. Keep the adapter only if it wins on held-out data.

Practical guidance and best practices

Pick a good starting configuration

• Start with a reduction factor of 16 and one adapter per block.
• Use GELU or ReLU in the bottleneck.
• Initialize the up projection to zeros or very small values.
• Use a learning rate in the range of 1e-4 to 1e-3 and tune it.
• Train the task head together with the adapters.

Decide where to place adapters

If memory is tight, place adapters only in the upper layers, where features are more task-specific. For domains that differ a lot from pre-training, place them in all layers. Compare positions on a validation set rather than relying on rules of thumb.

Control overfitting

Small datasets can overfit even with few parameters. Use weight decay, dropout inside the adapter, early stopping and cross-validation. On tabular data, k-fold splits give a more honest estimate than a single split.

Evaluate against strong baselines

Always compare with a tuned tree ensemble such as LightGBM or CatBoost, and with the frozen backbone used in-context. If the adapter does not beat both, it is not worth the complexity.

Version and document adapters

• Store the backbone version hash with every adapter.
• Record the training data snapshot and the metric on a fixed test set.
• Keep a registry, for example with MLflow, so you can roll back.
• Test that loading an adapter on the wrong backbone fails loudly.

Serving tips

• Load the backbone once and keep adapters in memory for fast swapping.
• Batch requests by adapter to avoid thrashing.
• Measure latency with and without the adapter so you know the true cost.
• Consider merging into a single model, or switching to LoRA, if latency becomes a blocker.

Combine with quantization

A frozen backbone can be stored in lower precision to save memory, while adapters stay in higher precision. Tools such as bitsandbytes support this pattern. Check quality after quantization, because small errors in the backbone can interact with the adapter.

Use a reproducible workflow

Fix random seeds, log hyperparameters, and use a standard framework such as PyTorch. Keep training scripts short and readable, so that a reviewer can see exactly what is trained and what is frozen.

A short checklist before training

• Confirm the backbone is in evaluation mode for layers such as dropout, unless you want them active.
• Print the trainable parameter count and check that it matches your expectation.
• Run one batch and confirm the loss is finite.
• Check that the output at step zero matches the backbone output.
• Fix the data split before any tuning begins.

A short checklist after training

• Evaluate on a test set that was never used for tuning.
• Compare against the frozen backbone and a tree ensemble.
• Save the adapter, the head, the config and the backbone version.
• Reload the adapter in a fresh process and confirm the metric is the same.
• Record latency and memory for the serving setup.

Everyday analogy for choosing a size

Imagine labeling boxes for a shipping line. A tiny sticker is cheap and quick but holds little information. A full printed sheet holds a lot but costs more and takes time to attach. The bottleneck size works the same way. Pick the smallest label that still carries what the task needs.

How adapters relate to model merging

Because adapters are separate files, teams sometimes ask whether two adapters can be averaged. Averaging can work when adapters were trained from the same starting point on related tasks, but it is not guaranteed. Test merged adapters on both tasks before trusting them. A learned fusion layer is usually safer than plain averaging.

Security and governance notes

• Treat adapter files as code-like artifacts. Load them only from trusted sources.
• Prefer safe tensor formats over pickled files when sharing.
• Keep an audit trail of which adapter served which request.
• Remove adapters trained on data you no longer have the right to use.
• Test fairness metrics per adapter, not only for the backbone.

Monitoring in production

• Track input drift per feature, since an adapter was tuned to one distribution.
• Alert when prediction confidence shifts sharply.
• Schedule periodic retraining when drift crosses a set threshold.

Data requirements

Adapters need fewer labeled examples than training from scratch, but they still need clean labels. A few hundred to a few thousand rows can be enough for simple tabular tasks. More complex shifts need more. Learning curves, which plot validation score against training size, show whether more data would help.

Common mistakes

• Forgetting to freeze the backbone. If gradients flow into the base weights, you are doing partial full fine-tuning and lose the memory and isolation benefits.
• Not training the head. A randomly initialized head with a frozen backbone and adapters that start at identity will give poor results.
• Using a tiny learning rate. Adapters start near zero, so they often need a larger rate than full fine-tuning.
• Too small a bottleneck. If the model underfits, increase r before changing anything else.
• Mixing adapter and backbone versions. An adapter is only valid for the exact backbone it was trained on.
• Skipping the baseline. Without a plain baseline, you cannot tell whether the adapter helped.
• Ignoring latency. Extra layers add time. Measure it early.
• Training on leaked data. In tabular work, features that encode the target or the split cause inflated scores. Check pipelines for leakage.
• Over-tuning on a small validation set. Use nested or repeated cross-validation when data is scarce.
• Stacking too many adapters. Each one adds cost and may interfere. Add them only when a test shows gain.
• Changing preprocessing between training and serving. Scalers, category encoders and missing-value rules must be identical.

Frequently asked questions

What are adapters in machine learning?

Adapters are small trainable modules inserted into a frozen pre-trained model so that it can learn a new task. They usually have a down projection, a non-linearity, an up projection and a skip connection. Only these modules and the task head are trained, which keeps cost and storage low.

How are adapters different from LoRA?

Both methods train a small number of new parameters, but they work differently. Adapters add new layers in the forward path, while LoRA adds a low-rank update to existing weight matrices. LoRA updates can be merged into the base model after training, so they add no serving latency, whereas adapters stay as extra layers.

Do adapters reach the quality of full fine-tuning?

On many tasks adapters come close to full fine-tuning while training only a few percent of the parameters. On tasks far from the pre-training data, a gap can remain. The only reliable way to know is to test both on a held-out set from your own data.

How many parameters does an adapter add?

A bottleneck adapter adds roughly two times the model width times the bottleneck size per module, plus biases. With a reduction factor of 16 and adapters in every layer, the total is usually a small share of the base model, often one to a few percent. The exact number depends on model width, depth and where you place the modules.

Can I use several adapters at once?

Yes. You can stack adapters, for example a domain adapter under a task adapter, or learn a fusion layer that mixes their outputs. This is useful for multi-task transfer. Test combinations carefully, because interference between adapters can reduce quality.

Do adapters prevent catastrophic forgetting?

They greatly reduce it because the original weights never change. Removing the adapter returns the exact original model. The new task skill lives only in the adapter, so old skills are safe as long as you keep the right adapter attached for each task.

Can adapters be used on tabular models?

Yes. Any transformer-style backbone can host adapters, including models pre-trained on tables. They let you specialize a tabular foundation model to a domain, such as a certain industry or data source, without retraining the backbone. Always compare against tree ensembles and the zero-shot model.

Do adapters slow down inference?

Slightly. Because adapters are extra layers, each forward pass does more work, and the extra depth adds some latency. The effect is usually modest, but in strict latency settings it can matter. Methods that can be merged into the base weights avoid this cost.

How do I choose the bottleneck size?

Start with a reduction factor around 16 and test smaller factors if the model underfits. A smaller reduction factor means a wider bottleneck and more capacity. Use validation performance and training curves to decide, and stop increasing the size when gains flatten.

When should I not use adapters?

Skip adapters when you have one task and unlimited compute, since full fine-tuning may be simpler. Also skip them when serving latency is critical and a mergeable method works as well. For very tiny tasks, a simpler option such as bias-only tuning or a plain linear head may be enough.

Glossary

• Adapter: A small trainable module added inside a frozen model.
• AdapterFusion: A method that learns to combine outputs from several adapters.
• Backbone: The large pre-trained network that supplies general features.
• Bottleneck: A narrow layer that reduces dimension before expanding again.
• Catastrophic forgetting: Loss of earlier skills when learning new ones.
• Fine-tuning: Further training of a pre-trained model on a target task.
• Foundation model: A large model pre-trained on broad data and reused for many tasks.
• Frozen weights: Weights that do not receive updates during training.
• GELU: A smooth activation function common in transformers.
• In-context learning: Predicting from examples placed in the input, without weight updates.
• Masked cell modeling: Pre-training by hiding table cells and predicting them.
• PEFT: Parameter-efficient fine-tuning, methods that train few parameters.
• Reduction factor: Model width divided by adapter bottleneck width.
• Residual connection: A path that adds a layer's input to its output.
• Task head: The output layer for a specific prediction task.
• Transfer learning: Reusing knowledge from one task or dataset on another.

Summary

Adapters are small bottleneck modules inserted into a frozen pre-trained model. They train only a few percent of the parameters, store as tiny files, and protect the original knowledge. Their main cost is a little extra inference latency, since they cannot be merged the way LoRA updates can.

For tabular foundation models, adapters offer a clean way to specialize a table-native backbone to a new domain, task or label set without retraining it. They fit multi-task and continual setups, and they pair well with in-context prediction as a way to go further when data allows. Test them against the zero-shot model and a tuned tree ensemble, version them with the backbone, and keep them only when they win. For related methods see prefix tuning, prompt tuning, IA3, BitFit and full fine-tuning, and read more at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: