IA3

Overview

IA3 stands for Infused Adapter by Inhibiting and Amplifying Inner Activations. It is a parameter-efficient fine-tuning method that keeps a pre-trained model frozen and learns a few small vectors. Each vector rescales an internal activation, either shrinking it (inhibiting) or enlarging it (amplifying). Because the learned objects are only vectors, IA3 trains far fewer parameters than most other methods, often a tiny fraction of one percent of the model.

IA3 was designed for few-shot fine-tuning of text models, and it is a core part of the T-Few recipe. Its learned scalings can be merged into the existing weights after training, so it adds no inference latency. It belongs to the PEFT family together with LoRA, adapters, prefix tuning, prompt tuning and BitFit. For tabular foundation models, IA3 is an attractive way to nudge a table-native backbone toward a domain with very little data and almost no storage.

What it is

IA3 multiplies certain internal activations of a transformer by learned vectors, element by element. The vectors are initialized to ones, so at the start the model behaves exactly as before. Training moves the entries up or down. An entry above one amplifies a feature. An entry below one inhibits it.

Picture a mixing desk in a recording studio. The band, which is the pre-trained model, is fixed. The engineer only moves the sliders. Each slider controls the volume of one channel. With a handful of sliders, the engineer can change the sound a lot without touching any instrument.

Which activations are rescaled

In a transformer block, IA3 learns three scaling vectors:

• One for the attention keys.
• One for the attention values.
• One for the hidden activations inside the feed-forward network.

Each vector has the size of the activation it scales. In an encoder-decoder model, the scalings are applied in both parts, and in the cross-attention too.

How small is it

Per layer, the number of new parameters is about the key width plus the value width plus the feed-forward width. For a model with a large number of layers, the total is typically a very small share of the weights. That is why IA3 is among the lightest PEFT methods.

No bottleneck, no low-rank matrix

IA3 has no inner layers and no low-rank factors. It only has scaling vectors. This is the reason it is so small, and also the reason it has limited capacity compared with methods that add matrices.

Mergeable by design

A scaling of an activation produced by a linear layer can be folded into that layer's weights, by scaling the rows or columns accordingly. After training, you can merge IA3 into the base model. The result has the same shape and speed as the original model.

Where it came from

IA3 was introduced in the 2022 paper "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning" by Liu and colleagues. The paper presented a recipe called T-Few. It combined IA3 with a pre-trained text-to-text model, a few additional loss terms designed for few-shot learning, and a short training schedule. The authors argued that in a few-shot setting, a small amount of fine-tuning can beat in-context learning with a much larger model, while costing less at inference time because long example prompts are not needed.

The paper sits in a line of work on small-footprint adaptation:

• Adapters (2019) added small bottleneck modules.
• Prefix tuning and prompt tuning (2021) learned continuous prompts.
• BitFit (2021) trained only bias terms.
• LoRA (2021) learned low-rank updates.
• IA3 (2022) trained scaling vectors.

The common goal in all of them is to cut storage and training cost. IA3 sits at the extreme end of the scale in terms of parameter count. It is available in the PEFT library and in the Hugging Face ecosystem. For the research context of reusing one backbone, see /model/research.

How it works

Step 1: Freeze the pre-trained model

Load a pre-trained transformer and set all weights to frozen. The scaling vectors will be the only trainable objects, together with an optional head.

Step 2: Create the scaling vectors

For each transformer layer, create three vectors:

• A key scaling vector, the size of the key projection output.
• A value scaling vector, the size of the value projection output.
• A feed-forward scaling vector, the size of the hidden feed-forward activation.

Initialize all entries to 1.0.

Step 3: Apply them during the forward pass

When the model computes keys, it multiplies them elementwise by the key vector. The same is done for values. In the feed-forward block, the hidden activation after the non-linearity is multiplied by the feed-forward vector before the output projection.

K' = l_k * K
V' = l_v * V
H' = l_ff * activation(W1 x)
output = W2 H'

Here l_k, l_v and l_ff are the learned vectors and the star is elementwise multiplication.

Step 4: Train only those vectors

Gradients flow through the frozen network to the scaling vectors. The optimizer updates only them. Because there are so few parameters, memory for optimizer state is negligible, although activations for the backward pass are still needed.

Analogy: think of a large choir with fixed singers. IA3 does not change who sings. It only adjusts the volume of each voice section. A few volume knobs can reshape the overall sound.

Step 5: Merge or keep separate

After training, you have two choices:

1. Merge the vectors into the weights for zero overhead.
2. Keep the vectors separate and swap them per task.

Keeping them separate enables multi-task serving from one backbone. Merging gives the fastest single-task model.

Step 6: Save a tiny file

The IA3 file holds only the scaling vectors. It can be tens or hundreds of kilobytes, even for a very large model.

Why scaling works

Pre-trained models already contain many useful features. A new task often needs a different mix of those features, not new features. Rescaling lets the model emphasize the useful ones and suppress the rest. This is a strong prior for few-shot learning, where there is not enough data to learn new features.

Where the key and value scalings matter

Scaling keys changes how strongly each feature contributes to the attention score, which changes where the model looks. Scaling values changes what information is passed on once attention picks a position. Scaling the feed-forward hidden units changes which stored patterns are active. Together they control looking, reading and recalling.

The T-Few recipe in short

The extra loss terms are specific to the T-Few setting. IA3 itself can be used without them.

Hyperparameters

Variants and relatives

Key concepts and terms

A worked example

The sketch below applies IA3-style scalings to a small attention layer and a feed-forward block in PyTorch. It is a teaching example.

import torch
import torch.nn as nn
import torch.nn.functional as F


class IA3Block(nn.Module):
    def __init__(self, d_model=64, d_ff=256):
        super().__init__()
        # frozen base weights
        self.q = nn.Linear(d_model, d_model)
        self.k = nn.Linear(d_model, d_model)
        self.v = nn.Linear(d_model, d_model)
        self.o = nn.Linear(d_model, d_model)
        self.w1 = nn.Linear(d_model, d_ff)
        self.w2 = nn.Linear(d_ff, d_model)
        for p in self.parameters():
            p.requires_grad = False
        # trainable scalings, initialized to ones
        self.l_k = nn.Parameter(torch.ones(d_model))
        self.l_v = nn.Parameter(torch.ones(d_model))
        self.l_ff = nn.Parameter(torch.ones(d_ff))

    def forward(self, x):
        q = self.q(x)
        k = self.k(x) * self.l_k
        v = self.v(x) * self.l_v
        a = F.scaled_dot_product_attention(q, k, v)
        x = x + self.o(a)
        h = F.gelu(self.w1(x)) * self.l_ff
        return x + self.w2(h)


blk = IA3Block()
train = [p for p in blk.parameters() if p.requires_grad]
print(sum(p.numel() for p in train))   # 384
total = sum(p.numel() for p in blk.parameters())
print(total)

Reading the numbers

• The trainable count is 64 plus 64 plus 256, so 384.
• The frozen weights of this toy block are about 49,000.
• So IA3 trains well under one percent of the block.

Training loop outline

opt = torch.optim.AdamW(train, lr=3e-3, weight_decay=0.0)
for step in range(100):
    x = torch.randn(8, 10, 64)
    target = torch.randn(8, 10, 64)
    loss = F.mse_loss(blk(x), target)
    opt.zero_grad()
    loss.backward()
    opt.step()

Merging the scalings

The key scaling can be merged into the key projection. If the key projection is K = W_k x + b_k, then a scaled key is l_k * K, which equals (diag(l_k) W_k) x + l_k * b_k. So you multiply each row of the weight and each bias entry by the matching scale. The same logic holds for values.

For the feed-forward scale, the vector multiplies the hidden activation, so it can be folded into the columns of the second weight matrix. After this, the model has the original architecture with new weights.

@torch.no_grad()
def merge(blk):
    blk.k.weight.mul_(blk.l_k[:, None])
    blk.k.bias.mul_(blk.l_k)
    blk.v.weight.mul_(blk.l_v[:, None])
    blk.v.bias.mul_(blk.l_v)
    blk.w2.weight.mul_(blk.l_ff[None, :])
    blk.l_k.fill_(1.0)
    blk.l_v.fill_(1.0)
    blk.l_ff.fill_(1.0)

After merging, the scaling vectors are reset to ones, and the block gives the same outputs as before merging.

Numeric intuition

Suppose a key vector has entries [0.8, -0.3, 1.2] and the learned scale is [1.5, 0.2, 1.0]. The scaled key is [1.2, -0.06, 1.2]. The first feature is amplified. The second is almost silenced. The third is unchanged. Attention scores now rely more on the first and third features.

Using a library

# Pseudocode shape of a typical workflow
base = load_pretrained("base-model-name")
cfg = IA3Config(target_modules=["k_proj", "v_proj", "ffn_out"],
                feedforward_modules=["ffn_out"])
model = wrap_with_peft(base, cfg)
train(model, train_data, lr=3e-3)
model.save_adapter("ia3_task_a")

Module names depend on the architecture, so check the model definition and the PEFT documentation. Libraries such as TRL can drive training.

Strengths and limitations

Strengths

• Extremely small number of trainable parameters.
• Tiny files per task, easy to store and ship.
• No inference latency after merging.
• Strong prior for few-shot learning, because it reuses existing features.
• Low memory for optimizer state.
• Simple to implement and to reason about.
• Initialization at ones preserves the base model at step zero.
• Works with quantized backbones, since the scalings are small.

Limitations

• Limited capacity. It can only reweight existing features, not create new ones.
• May fall behind LoRA and full fine-tuning when the target task is far from pre-training.
• Needs a reasonably strong pre-trained model to work well.
• Results can be sensitive to which modules are scaled.
• Less common in the community than LoRA, so tooling and examples are thinner.
• Best evidence comes from text-to-text models and few-shot settings.
• Merging changes the weights, so keep a copy of the original if you want to undo it.

When capacity runs out

If training loss plateaus high and validation scores are far below those of a larger-capacity method, the scaling vectors may be too limited. Options include adding LoRA on top, scaling more modules, or moving to full fine-tuning or continued pre-training.

How it compares with alternatives

Reading the table

IA3 and BitFit are the lightest. LoRA and its relatives sit in the middle with more capacity and good tooling. Adapters and prompt-based methods are modular but add inference cost. Full fine-tuning has the highest ceiling at the highest price.

IA3 versus LoRA

LoRA adds a low-rank matrix update to chosen weight matrices. IA3 only rescales activations. LoRA has more capacity and is the default in many stacks. IA3 has fewer parameters and is a good fit for few-shot tasks and tight storage budgets. A practical approach is to try both on a validation set.

IA3 versus in-context learning

In-context learning places examples in the prompt, which makes every request longer and more costly. IA3 spends a small training cost once, and then each request is short. In the few-shot setting studied in the T-Few paper, this was reported to be both cheaper at inference time and more accurate than in-context learning with a larger model.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types, missing cells and often no useful column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large numbers of synthetic or real tables, and they predict on new tables by reading labeled rows as context. IA3 gives a very light way to adapt such models. See /model/architecture for how table-native backbones are structured.

Reweighting features in a table

The central idea of IA3 is to emphasize some internal features and suppress others. In a tabular model, internal features carry information about columns, value ranges and interactions. A new domain may need a different emphasis. For example, in a credit table, income-related features may matter more than in a retail table. Scaling vectors on attention keys and values can shift the model's attention toward the features that matter for the domain.

Row and column attention

Tabular transformers attend across rows and across columns. IA3 vectors can be placed in either path. Scaling keys and values in the column path changes how features interact. Scaling them in the row path changes how records are compared. Because the vector count is tiny, you can try both paths cheaply and keep the better one.

Mixed types and missing values

Cell encoders map numbers, categories, dates and text into a shared space, with a special marker for missing values. IA3 acts after this step, on internal activations. It can down-weight activations that react to noisy or misleading types in a domain, and up-weight those that help. The encoders remain frozen.

No column names

With no headers, a model must infer meaning from values and structure. Text-aware models such as TabSTAR use names when present. IA3 can help a header-free domain by emphasizing features that capture typical value patterns, though this has to be tested.

Few-shot behavior on small tables

Many real tables are small. This matches the regime where IA3 was designed: few labeled examples, a strong pre-trained prior, and a need to avoid overfitting. Tabular foundation models already do well on small tables through in-context prediction. IA3 can add a small, cheap domain adjustment on top when you have a few hundred labeled rows from a stable source.

Pre-training on synthetic and real tables

Synthetic pre-training teaches broad structure, and real tables add realistic semantics. IA3 sits after pre-training as a thin layer of specialization. Because it only reweights features that exist, it depends on the pre-trained model having covered the needed patterns. Check results against the plain in-context model and tuned trees such as XGBoost. Evaluation suites are summarized at /model/benchmarks.

Continual learning

For new domains over time, train a new IA3 vector set per domain, and leave the backbone alone. Files are tiny, so thousands of domains can be stored. This avoids forgetting without replay (experience replay) or weight constraints (EWC). At serve time you still need a way to select the right set, or to average sets for related domains.

Imputation and anomaly detection

For imputation, a backbone pre-trained with masked cell modeling predicts hidden cells. IA3 can bias it toward the patterns of a table family, such as sensor data with block-wise gaps. For anomaly detection, scaling can sharpen the model's sense of normal rows for a system. See also imputation networks.

Sector and semantic classification

Assigning a sector label to a table or entity needs semantic cues from names and values. IA3 plus a semantic head can specialize a backbone for a label set with very little storage. If labels change, retrain the vectors and head only.

Honest caveats

The strongest evidence for IA3 is on text models in a few-shot setting. Applying it to tabular backbones is an extension of the same mechanism. Gains are not guaranteed. Test on held-out folds, and compare with the frozen model, LoRA and a tuned tree ensemble.

Practical guidance and best practices

A sensible order of experiments

1. Score the frozen backbone with in-context examples.
2. Train IA3 on keys, values and feed-forward activations.
3. Try fewer or more modules to see how capacity changes the score.
4. Compare with LoRA and BitFit.
5. Compare with LightGBM or CatBoost tuned on the same split.
6. Keep the lightest method that meets your target.

Choose modules with care

Scale keys and values in attention and the hidden activation in the feed-forward block, as in the original design. If you scale only some of them, capacity falls. If the architecture has unusual layers, check the library mapping so that the right projections are targeted.

Use a higher learning rate

Scaling vectors start at one and need to move. A rate higher than for full fine-tuning is typical. Watch for instability, and use warm-up if needed.

Avoid weight decay on scalings

Weight decay pulls parameters toward zero, which would shrink the scalings and silence features. If you use weight decay, apply it toward one, or turn it off for these vectors.

Validate properly

On tabular data use k-fold cross-validation with leakage-safe preprocessing. Fit scalers and encoders inside each fold. Report mean and spread across folds and seeds.

Merge for production when possible

If you serve a single task, merge the scalings into the weights for zero overhead. Keep the unmerged version and the original weights in the registry. If you serve many tasks, keep vectors separate and swap them.

Version and track

• Store the vectors with the exact backbone version.
• Log hyperparameters and metrics with a tracker such as MLflow.
• Add a test that refuses to load vectors on the wrong backbone.

Combine with quantization

A quantized backbone, produced with tools such as bitsandbytes, can host IA3 vectors in higher precision. Merging into a quantized layer needs care, so test outputs before and after merging.

Pre-flight checklist

• Print the trainable parameter count and verify it equals the vector sizes.
• Confirm that all scalings start at one.
• Check that the first-step output equals the base model output.
• Overfit a tiny batch to prove that the vectors can learn.
• Fix the evaluation split before tuning.

Everyday analogy for choosing methods

If you need to adjust a recipe slightly, you can change the amounts of existing ingredients (IA3), add a pinch of one new ingredient (LoRA), or rewrite the recipe (full fine-tuning). Start with the lightest change and move up only if the dish is not right.

Debugging a run that does not learn

1. Confirm that the vectors are registered parameters and are in the optimizer.
2. Print the gradient norm of each vector after one backward pass.
3. Raise the learning rate by a factor of ten and watch the loss.
4. Check that weight decay is off for the vectors.
5. Confirm that the target modules were actually matched by name.
6. Check that labels and inputs are aligned.

What to log

• Which modules were scaled and how many parameters were trained.
• Learning rate, seed and split identifier.
• The distribution of learned scales per layer, such as the minimum, median and maximum.
• Held-out metric and its spread across seeds.

Reading the learned scales

After training, plot the learned values per layer. Entries far from one show where the model changed most. If almost all entries stay near one, the task was close to the pre-training distribution, or the learning rate was too low. If many entries are near zero, the method may be silencing features, which can signal overfitting on a tiny set.

Governance

Scaling files are small, which makes them easy to copy around. Treat them as model artifacts: control access, track lineage, and record the data used to train them. Remove vectors trained on data you are no longer allowed to use.

Common mistakes

• Initializing scalings at zero. That would silence the activations. Initialize at one.
• Applying weight decay to the vectors. It pushes them toward zero and harms the model.
• Scaling the wrong modules. Check that keys, values and the feed-forward hidden layer are the ones targeted.
• Using too low a learning rate. The vectors need room to move.
• Expecting new features. IA3 only reweights existing ones.
• Merging without a backup. Keep the original weights if you may need to undo the change.
• Testing on one seed. Few-shot results vary, so use several seeds.
• Skipping baselines. Compare with the frozen model, LoRA and a tuned tree ensemble.
• Leaking data in tabular tasks. Fit all preprocessing inside the training fold.
• Reusing vectors across backbones. They are tied to the exact model they were trained on.
• Assuming it is best for every task. For large domain shifts, higher-capacity methods usually do better.

Frequently asked questions

What does IA3 stand for?

IA3 stands for Infused Adapter by Inhibiting and Amplifying Inner Activations. The name describes how it works: small learned vectors multiply internal activations, making some smaller and some larger. The method keeps the pre-trained model frozen and trains only those vectors.

How many parameters does IA3 train?

It trains very few. For each layer it learns three vectors, one for keys, one for values and one for the feed-forward hidden activation. Across a whole model this is usually a tiny fraction of one percent of the weights, which makes IA3 one of the lightest fine-tuning methods.

Does IA3 add inference latency?

No, once the scalings are merged into the base weights, the model has the same architecture and speed as the original. If you keep the vectors separate to swap tasks, there is a very small extra cost from the elementwise multiplications.

How is IA3 different from LoRA?

LoRA learns low-rank matrices that are added to weight updates, which gives more capacity. IA3 only learns scaling vectors that reweight activations, which gives fewer parameters and lower capacity. Both can be merged into the base model, so both avoid serving overhead.

When is IA3 a good choice?

IA3 is a good choice when you have few labeled examples, a strong pre-trained model, a tight storage budget, and a need for zero latency overhead. It is especially suited to few-shot settings. For large domain shifts or complex tasks, a higher-capacity method may work better.

What is T-Few?

T-Few is a few-shot fine-tuning recipe that combines IA3 with a pre-trained text-to-text model and extra loss terms. It was presented to show that a small amount of parameter-efficient fine-tuning can beat in-context learning in the few-shot setting while costing less at inference time.

Can IA3 be used on tabular models?

The mechanism applies to any transformer-style backbone, including those pre-trained on tables. IA3 can reweight internal features toward a domain with very little data and storage. As with any adaptation, compare with the zero-shot model, LoRA and tuned tree ensembles before adopting it.

Why initialize the scaling vectors at one?

Multiplying by one leaves an activation unchanged, so the model starts exactly as the pre-trained model. Training then moves entries away from one only where it helps. Starting at zero would erase the activations and break the model.

Can I combine IA3 with LoRA?

Yes, you can train both, with LoRA adding low-rank updates and IA3 adding scalings. This increases capacity at a small extra cost. Test the combination against each method alone, because gains depend on the task and the data size.

Does IA3 prevent forgetting?

Because the base weights are frozen during training, the original knowledge is preserved, and you can restore the original behavior by resetting the scalings to one. After you merge, the weights change, so keep a copy of the original if you need to roll back.

Glossary

• Activation: The output of a layer inside a neural network.
• Amplify: Increase the size of an activation.
• Backbone: The large pre-trained model kept frozen.
• Elementwise product: Multiplying two vectors entry by entry.
• Feed-forward network: The per-position network in a transformer block.
• Few-shot learning: Learning from a small number of labeled examples.
• Foundation model: A large model pre-trained on broad data for reuse.
• IA3: A PEFT method that learns scaling vectors for keys, values and feed-forward activations.
• In-context learning: Predicting from examples in the input without weight updates.
• Inhibit: Reduce the size of an activation.
• Masked cell modeling: Pre-training by hiding table cells and predicting them.
• Merging: Folding learned changes into the base weights.
• PEFT: Parameter-efficient fine-tuning.
• Scaling vector: A learned vector that multiplies an activation.
• T-Few: A few-shot recipe built around IA3.

Summary

IA3 freezes a pre-trained model and learns small vectors that rescale attention keys, attention values and feed-forward activations. It trains an extremely small number of parameters, stores in tiny files, and can be merged into the weights for zero inference overhead. It shines in few-shot settings and when storage is tight. Its limits are low capacity and thinner evidence outside text models.

For tabular foundation models, IA3 offers a cheap way to shift emphasis among internal features toward a domain, a source or a label set, while the backbone stays shared. It suits small tables, per-domain files and continual setups. Always check it against the in-context baseline, LoRA, adapters and a tuned tree ensemble. For related methods see BitFit, prefix tuning, prompt tuning and full fine-tuning, and read more at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: