BitFit

Overview

BitFit is a sparse fine-tuning method that updates only the bias terms of a pre-trained model, plus the task head. Every other weight stays frozen. The name comes from "bias-terms fine-tuning." In a typical transformer, biases make up a very small share of all parameters, often well under one percent, so BitFit is among the cheapest ways to adapt a model.

The method is useful when you have a small amount of labeled data, a strong pre-trained backbone, and a tight storage or memory budget. It is simple to implement, it adds no new layers, and it adds no inference cost. It sits in the PEFT family alongside LoRA, adapters, IA3, prefix tuning and prompt tuning. Its capacity is limited, so it works best when the new task needs only a small shift in what the model already knows. For tabular foundation models, BitFit is a minimal baseline to test before heavier adaptation.

What it is

Most layers in a neural network compute an output as a weight matrix times the input, plus a bias vector. The bias is a learned offset that shifts the output. BitFit trains only those offsets. The large weight matrices are not touched.

Picture a large pipe organ. The pipes and the keyboard are fixed. A technician can only turn small tuning screws that shift each pitch slightly. With the right turns, the organ can sound better in a new hall, even though no pipe is replaced.

Which parameters are trained

Some models drop bias terms in certain layers. In those, BitFit has less to train. In a model with no biases at all, BitFit cannot be applied in its pure form.

How small is it

For a standard transformer, bias vectors have one entry per output unit. A layer with a width of 768 and a feed-forward width of 3,072 has a few thousand bias entries per sub-layer, compared with millions of weight entries. Across the whole model, biases are commonly around 0.05 to 0.1 percent of all parameters, though the exact share depends on the architecture.

Why it can work

Pre-trained models already contain a rich set of features. Fine-tuning often does not need to create new features. It only needs to expose the right ones to the output layer. Changing biases shifts activation thresholds, which changes which units fire, and that can be enough to reroute information. The BitFit authors argued that fine-tuning is largely about exposing knowledge that the model already holds.

Additive, not multiplicative

BitFit shifts activations by adding offsets. IA3 scales activations by multiplying. Both are cheap, and they can be combined, though they act in different ways.

Where it came from

BitFit was proposed in 2021 in the paper "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models" by Ben Zaken, Ravfogel and Goldberg. The paper evaluated the idea on masked language models of the BERT family and reported that tuning only bias terms came close to full fine-tuning on a set of language understanding benchmarks, especially with small and medium training sets. With large training sets, the gap to full fine-tuning grew.

The work arrived in the middle of a burst of ideas about small-footprint adaptation:

• Adapters (2019), which add small bottleneck modules.
• Prefix tuning and prompt tuning (2021), which learn continuous prompts.
• LoRA (2021), which learns low-rank weight updates.
• IA3 (2022), which learns scaling vectors.

BitFit is notable for its simplicity: it needs no new modules and almost no code. It also gave a clean insight about what fine-tuning changes. It is supported in the PEFT ecosystem and is easy to reproduce by hand in PyTorch by setting gradients on or off for named parameters. For the research context of reusing a single backbone, see /model/research.

How it works

Step 1: Load a pre-trained model

Start with a pre-trained network. It could be a language model, a vision transformer or a model pre-trained on tables. Check that the layers have bias terms.

Step 2: Freeze everything

Set all parameters so that they do not receive gradients.

Step 3: Unfreeze biases and the head

Loop over the named parameters. For each one whose name indicates a bias, enable gradients. Add a task head, such as a linear classifier, and enable gradients for it too.

for name, p in model.named_parameters():
    p.requires_grad = name.endswith(".bias")

for p in head.parameters():
    p.requires_grad = True

Analogy: you lock every knob on a control panel, then unlock only the small trim dials. You can still change how the machine behaves, but only within a narrow range.

Step 4: Train with a standard loss

Run batches through the model, compute the loss, and backpropagate. Gradients are computed for the whole network where needed, but updates apply only to the unfrozen parameters. The optimizer state is kept only for these few tensors.

Step 5: Choose a learning rate

Because few parameters are trained, a higher learning rate than in full fine-tuning is often used. The original paper explored learning rates in a range typical for such methods. Tune it on validation data.

Step 6: Save a tiny file

Save the trained biases and the head. You can store them as a small delta from the base model. Loading the delta onto the base model restores the tuned model.

Step 7: Serve

Because BitFit changes only parameter values and adds no layers, the tuned model has the same shape and speed as the original. You can keep one base model plus a small bias delta per task and apply the delta when needed.

Subsets of biases

The paper also studied training only some biases. The biases of the query and the first feed-forward layer, for example, carried much of the benefit in their experiments. This suggests that even fewer parameters may suffice for some tasks. Such sparse variants can be explored when the budget is extreme.

Why the head matters

BitFit without a task head cannot produce task outputs, since a new task needs a new output layer. Linear probing trains only the head. BitFit trains the head plus the biases, and it usually beats probing alone because it can change the features that feed the head.

Training cost

Memory use is lower than full fine-tuning because the optimizer keeps no states for the frozen matrices. Compute for the forward and backward pass is still close to full fine-tuning, because gradients must flow through the network to reach the early biases. So BitFit saves memory and storage more than it saves time.

Variants and relatives

Key concepts and terms

A worked example

This PyTorch sketch applies BitFit to a small classifier. It freezes everything except the biases and the new head.

import torch
import torch.nn as nn


class Backbone(nn.Module):
    def __init__(self, d_in=32, d=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_in, d), nn.ReLU(),
            nn.Linear(d, d), nn.ReLU(),
            nn.Linear(d, d), nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)


backbone = Backbone()
head = nn.Linear(64, 3)

for name, p in backbone.named_parameters():
    p.requires_grad = name.endswith("bias")
for p in head.parameters():
    p.requires_grad = True

params = [p for p in list(backbone.parameters()) + list(head.parameters())
          if p.requires_grad]
n_train = sum(p.numel() for p in params)
n_all = sum(p.numel() for p in list(backbone.parameters()) + list(head.parameters()))
print(n_train, n_all)

opt = torch.optim.AdamW(params, lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
x = torch.randn(128, 32)
y = torch.randint(0, 3, (128,))
for step in range(100):
    logits = head(backbone(x))
    loss = loss_fn(logits, y)
    opt.zero_grad()
    loss.backward()
    opt.step()

Counting parameters by hand

• First layer: 32 times 64 weights, so 2,048, and 64 biases.
• Second layer: 64 times 64 weights, so 4,096, and 64 biases.
• Third layer: 64 times 64 weights, so 4,096, and 64 biases.
• Backbone biases: 192. Backbone weights: 10,240.
• Head: 64 times 3 weights plus 3 biases, so 195.
• Trained in BitFit: 192 plus 195, which is 387.
• Total parameters: 10,240 plus 192 plus 195, which is 10,627.

So about 3.6 percent of this toy model is trained. In a real transformer, the share is much smaller, because weight matrices grow with the square of the width and biases grow only linearly.

Saving the delta

base_state = {k: v.clone() for k, v in backbone.state_dict().items()}
# ... after training ...
delta = {k: v - base_state[k]
         for k, v in backbone.state_dict().items() if k.endswith("bias")}
torch.save({"delta": delta, "head": head.state_dict()}, "bitfit_task_a.pt")

To restore, load the base model, add the delta to the matching biases, and load the head.

Numeric intuition

Take a unit that computes relu(w · x + b). Suppose w · x is 0.4 for typical inputs and the bias is -0.5, so the unit outputs zero. After tuning, the bias becomes -0.1. The unit now outputs 0.3. No weight changed, yet a feature that was silent is now active. This is how bias updates expose existing knowledge.

Using a library

# Pseudocode shape of a typical workflow
model = load_pretrained("base-model-name")
for n, p in model.named_parameters():
    p.requires_grad = "bias" in n
model.classifier.requires_grad_(True)
train(model, train_data, lr=1e-3)
save_delta(model, "bitfit_task_a")

Some libraries expose a bias-only option in their PEFT config. Check the documentation of the PEFT package for the exact setting names.

Strengths and limitations

Strengths

• Extremely small trainable set, often well under one percent.
• No new layers, so no inference overhead.
• Very simple. A few lines of code in any framework.
• Tiny delta files, easy to store and ship.
• Works well in small and medium data regimes on language understanding tasks, according to the original study.
• Lower memory for optimizer states than full fine-tuning.
• Keeps the backbone intact, so you can recover it by removing the delta.

Limitations

• Low capacity. Only offsets change, so large shifts in behavior are hard.
• The gap to full fine-tuning grows with more training data.
• Compute savings are small, since backward passes still traverse the network.
• Not applicable to models without bias terms. Several modern language models drop biases in many layers.
• Less popular than LoRA, so there are fewer recipes and examples.
• Sensitive to learning rate.
• Evidence is strongest on encoder language models, not on every domain.

The no-bias problem

Some recent architectures remove biases from linear layers to simplify training or save parameters. For those, BitFit has few or no targets. In such cases use IA3, LoRA or normalization-parameter tuning. Always inspect the parameter names of your model before choosing a method.

How it compares with alternatives

Reading the table

BitFit and linear probing are the cheapest. BitFit is a step above probing because it can change features. LoRA and adapters add real capacity. Full fine-tuning has the highest ceiling.

BitFit versus LoRA

LoRA can express changes to weight matrices, which gives it more power on hard tasks. BitFit only shifts offsets. LoRA has become the default in many large language model workflows, while BitFit is a good minimal baseline and a fit for very small budgets.

BitFit versus full fine-tuning

Full fine-tuning can reach higher accuracy with plenty of data, but it needs a full model copy per task and carries a forgetting risk. BitFit needs only a small delta and keeps the backbone intact. On small data, the two can be close.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types, missing cells and often no useful column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large numbers of synthetic or real tables, and predict on new tables by reading labeled rows as context. BitFit provides a very small baseline for adapting such models. See /model/architecture for how table-native backbones are built.

Why a bias-only baseline is useful for tables

Many real tables are small, and many domains shift only slightly from what the backbone saw in pre-training. In that case, large changes are unnecessary and risky. Shifting activation thresholds, which is what bias updates do, may be enough to expose the features that matter for the domain. A bias-only run is a quick way to learn how much adaptation a domain needs. If BitFit gives most of the gain, heavier methods are unnecessary.

Row and column structure

Tabular transformers have blocks for attention across columns and across rows. Each block has linear layers with biases. BitFit can tune all of them, or only those in one path. Tuning column-path biases changes how features respond. Tuning row-path biases changes how records are compared. Because the parameter count is tiny, trying subsets is cheap.

Mixed types and missing values

Cell encoders for numbers, categories, dates and text have their own linear layers, often with biases. A bias shift in a numeric encoder moves the representation of values, which can recenter inputs for a domain whose scale differs from pre-training. A bias in the missing-value path can change how the model treats absent cells. This is one of the more natural uses of additive offsets in tables. Check whether a given model keeps biases in its encoders.

No column names

Without headers, the model depends on value patterns. When names exist, text-aware models such as TabSTAR can use them. A bias-only update can recenter internal activations for a domain where names are missing, though it cannot create new naming knowledge.

In-context learning and BitFit

Some tabular foundation models predict by in-context learning: they read labeled rows and query rows in one pass, with no gradient steps. BitFit is a light way to go beyond pure in-context use. A short bias-only run on a few hundred labeled rows can shift the model toward a domain, and the in-context mechanism still works afterwards. Compare against the plain in-context model and a tuned tree ensemble such as XGBoost, as summarized in /model/benchmarks.

Pre-training on synthetic and real tables

Synthetic pre-training teaches generic structure, and real tables add semantics. BitFit comes after both. Since it can only offset existing features, it depends on the pre-trained model having covered what the domain needs. When the domain is far from anything the model saw, expect BitFit to underperform higher-capacity methods.

Continual learning

For new domains over time, store one small bias delta per domain and leave the backbone untouched. This isolates domain knowledge and avoids forgetting without replay (experience replay) or weight constraints (EWC). You need a way to pick the right delta at serve time. Because deltas are just offsets, averaging deltas from related domains is a simple experiment, though it must be tested.

Imputation and anomaly detection

For imputation, a backbone trained with masked cell modeling predicts hidden cells. A bias delta can recenter predictions for a table family, such as sensor logs with a known offset. For anomaly detection, a bias shift can tune what counts as typical for a system. See also imputation networks.

Sector and semantic classification

A bias delta plus a semantic head can specialize a backbone for a label set such as industry sectors. If labels change, retrain the delta and head only.

Honest caveats

BitFit was studied on masked language models. Its use on tabular backbones is an extension. Gains are not guaranteed, and some table-native architectures may use few biases. Treat it as a first, cheap experiment, and move to adapters or LoRA if it falls short.

Practical guidance and best practices

A sensible order of experiments

1. Score the frozen backbone, in context if the model supports it.
2. Train a linear probe on frozen features.
3. Train BitFit with the same head.
4. Train LoRA or IA3 if BitFit is not enough.
5. Compare with a tuned LightGBM or CatBoost model.
6. Keep the cheapest method that meets your target.

Check that biases exist

Print the parameter names and shapes before training. Count the bias entries. If there are too few, choose a different method. Do not assume all layers have biases.

Hyperparameters

Validate properly

On tabular data use k-fold cross-validation with leakage-safe preprocessing. Fit scalers and encoders inside each fold. Report mean and spread.

Store deltas, not copies

Keep one base model and a small delta per task. Version both. Include the base model hash in each delta file. Add a check that refuses to load a delta on the wrong base.

Track and govern

• Log hyperparameters and metrics with a tracker such as MLflow.
• Record the training data snapshot.
• Control access to delta files if they were trained on sensitive data.
• Delete deltas when the data right expires.

Combine with other methods

BitFit can be used together with LoRA or IA3. Adding bias training to LoRA is a small extra cost. Test the combination, since gains are task dependent.

Quantized backbones

If the backbone is stored in low precision with tools such as bitsandbytes, keep the biases in higher precision so that they can be trained. Check how the library treats biases in quantized layers.

Pre-flight checklist

• Print the trainable parameter count and confirm it equals biases plus head.
• Confirm that no weight matrix has gradients.
• Check that the loss is finite on the first batch.
• Overfit a tiny batch to prove that the setup can learn.
• Fix the evaluation split before tuning.

Everyday analogy for choosing a method

If a shirt is a little loose, you can use a belt (BitFit), take in a seam (LoRA), or sew a new shirt (full fine-tuning). Start with the belt. If it does not fit, move to the next option.

Debugging a run that does not learn

1. Confirm that the biases are in the optimizer.
2. Print gradient norms for the biases and the head.
3. Raise the learning rate.
4. Check that labels and inputs are aligned.
5. Confirm that the model actually has bias terms in the layers you expect.
6. Try tuning only the head first, to check the data pipeline.

Debugging a run that overfits

• Lower the learning rate or the number of epochs.
• Add early stopping on a validation fold.
• Use more data or stronger cross-validation.
• Look for duplicate rows shared between train and validation sets.
• Average deltas from several seeds.

What to log

• Number of bias entries trained and number of head parameters.
• Learning rate, seed and split identifier.
• Training and validation loss per epoch.
• Held-out metric and its spread across seeds.
• Size of the saved delta file.

Inspecting what changed

After training, compute the size of the change in each layer's bias, for example the mean absolute difference from the base value. Layers with large changes show where the model needed to shift. If changes are tiny everywhere, the task was close to pre-training or the learning rate was too low. If changes are large in the earliest layers only, the input scale of the domain may differ from pre-training, and a normalization fix on the input could help.

Where BitFit is a poor fit

• Tasks that need new knowledge not present in the backbone.
• Large datasets where higher-capacity methods can use the extra data.
• Models with few or no bias terms.
• Settings where you must merge many tasks into one set of weights without interference.

Where BitFit is a good fit

• Quick feasibility checks on a new dataset.
• Tight storage budgets with many small tasks.
• Backbones that must stay almost unchanged for audit reasons.
• Few labeled rows from a stable source.

Common mistakes

• Assuming every model has biases. Some layers have none, so BitFit may do almost nothing.
• Forgetting the head. A new task needs a trainable output layer.
• Leaving weight matrices trainable. Then you are doing full fine-tuning in disguise.
• Using too low a learning rate. Few parameters often need larger steps.
• Expecting full fine-tuning quality on large datasets. The gap grows with data size.
• Skipping the linear probe baseline. It shows how much BitFit adds.
• Weight decay on biases. It pushes offsets toward zero and wastes the method.
• Saving whole models. Store the delta instead.
• Data leakage in tabular tasks. Fit all preprocessing inside the training fold.
• Single-seed claims. Report several runs.
• Loading a delta on the wrong base. Always check the base version.

Frequently asked questions

What is BitFit?

BitFit is a fine-tuning method that updates only the bias terms of a pre-trained model, along with the task head, while all other weights stay frozen. It trains a very small share of the parameters and adds no layers. It was proposed as a simple, sparse alternative to full fine-tuning.

How much of a model do biases make up?

In standard transformers, biases are a very small share of the parameters, often well under one percent. Weight matrices grow with the square of the width, while biases grow only linearly. The exact share depends on the architecture and on which layers keep biases.

Does BitFit add inference cost?

No. BitFit adds no new layers and changes only parameter values, so the tuned model has the same shape and speed as the original. You can store the tuned biases as a small delta and apply it to the base model when needed.

How does BitFit compare with LoRA?

LoRA learns low-rank updates to weight matrices and has more capacity, so it usually does better on hard tasks and large datasets. BitFit only shifts offsets and is simpler and smaller. BitFit is a good first baseline or a fit for very small budgets.

When does BitFit work best?

It tends to work best with a strong pre-trained model, a small or medium labeled dataset, and a task that is close to what the model already knows. With large datasets or large domain shifts, higher-capacity methods usually win.

Why can changing only biases help?

Biases set the thresholds at which units respond. Moving a threshold can turn a silent feature on or a noisy feature off, which changes how information flows to the output. If the useful features already exist in the model, such shifts can be enough.

Can BitFit be used on tabular models?

The method applies to any model with bias terms, including transformers pre-trained on tables. It is a cheap way to test how much adaptation a domain needs. Compare it with the frozen model, LoRA and a tuned tree ensemble before adopting it.

What if my model has no bias terms?

Then BitFit has little or nothing to train. Use another method such as IA3, LoRA, adapters or tuning of normalization parameters. Inspect the parameter names to see which layers have biases before you plan the experiment.

Does BitFit save training time?

It saves memory and storage more than time. The optimizer needs states only for the biases, but the backward pass still runs through the network to reach early layers. Total compute is therefore close to full fine-tuning for the same number of steps.

Can I combine BitFit with other methods?

Yes. Training biases together with LoRA or IA3 is a common experiment. The extra cost is small. Test the combination against each method alone, because the benefit depends on the task and on data size.

Glossary

• Backbone: The large pre-trained model kept mostly frozen.
• Bias term: A learned offset added to a layer's output.
• Capacity: How much a method can change a model's behavior.
• Delta: The difference between tuned and base parameters.
• Fine-tuning: Further training of a model on a target task.
• Foundation model: A large model pre-trained on broad data for reuse.
• Frozen weights: Weights that receive no updates.
• In-context learning: Predicting from examples in the input without weight updates.
• Layer normalization: A step that rescales activations in a layer.
• Linear probing: Training only a linear head on frozen features.
• Masked cell modeling: Pre-training by hiding table cells and predicting them.
• PEFT: Parameter-efficient fine-tuning.
• Sparse fine-tuning: Updating only a small subset of parameters.
• Task head: The output layer for a specific task.
• Threshold: The input level at which a unit begins to respond.

Summary

BitFit fine-tunes a pre-trained model by updating only its bias terms and a task head. It trains a tiny share of the parameters, adds no layers, adds no inference cost and produces small delta files. It works best with strong backbones, small or medium datasets and tasks close to pre-training. Its limits are low capacity, a growing gap on large datasets, and no use on models without biases.

For tabular foundation models, BitFit is a minimal, cheap baseline for adapting a table-native backbone to a domain, a source or a label set. It also helps measure how much adaptation a domain really needs. Compare it with the frozen in-context model, LoRA, IA3, adapters and a tuned tree ensemble. For related methods see prefix tuning, prompt tuning and full fine-tuning, and read more at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: