PEFT

Overview

PEFT is a Python library from the Hugging Face ecosystem that implements parameter-efficient fine-tuning. Instead of updating every weight in a large model, PEFT freezes the base model and trains a small number of extra parameters, such as low-rank matrices in LoRA, virtual tokens in prompt tuning or prefix tuning, or tiny scaling vectors in IA3. The result is a small file, often a few megabytes, that can be loaded on top of the same base model to change its behavior.

The library matters because it makes fine-tuning cheap and portable. Training needs far less memory, since the optimizer only tracks the small set of trainable weights. Storage is small, so one base model can serve many tasks by swapping adapters. PEFT plugs into the usual Transformers training flow, and trainers in libraries such as TRL and frameworks such as Axolotl use it under the hood. This article explains the idea with everyday analogies, shows code, lists strengths and limits, and ends with how the approach relates to models that learn from raw tables, as covered in /model/research.

What it is

PEFT stands for parameter-efficient fine-tuning. It names both a family of techniques and the library that implements them. The library takes a pretrained model, wraps it with a chosen method, marks which parameters are trainable, and gives you back a model that behaves like the original in code, so existing training scripts still work.

What "parameter-efficient" means

A large language model may have billions of weights. Full fine-tuning updates all of them, which needs memory for the weights, the gradients and the optimizer state. Parameter-efficient methods update well under one percent of the weights in many setups, or sometimes only a tiny fraction of that. The frozen weights are only read, not updated.

An analogy

Think of a large reference book that you do not want to reprint. Full fine-tuning reprints the whole book with your edits. PEFT keeps the book as is and adds a small set of sticky notes and a few bookmarks. Anyone who has the book can use your notes. Someone else can use different notes on the same copy. The notes are cheap to make, easy to carry and simple to remove.

Method families in the library

Other pages cover these in detail: adapters, BitFit, DoRA, rsLoRA and QLoRA.

What it is not

PEFT is not a trainer by itself, though it works with trainers. It is not a quantization library, although it works with bitsandbytes for low-bit weights. It is not a serving system, although adapters can be merged or loaded by servers such as vLLM.

Where it came from

The idea of adding small trainable modules to a frozen network is older than large language models. Researchers working on transfer learning showed that small adapter layers inserted into a pretrained network could match full fine-tuning on many tasks while updating a small part of the weights. See transfer learning.

As language models grew, the cost of full fine-tuning became a barrier. Several lines of work followed:

• Adapter layers for transformers, which insert small bottleneck modules.
• Prefix tuning and prompt tuning, which learn virtual tokens and leave the model untouched.
• LoRA, introduced in 2021, which adds a low-rank update to weight matrices and can be merged back after training.
• QLoRA, introduced in 2023, which trains LoRA adapters on a base model stored in 4-bit form, so large models fit on a single GPU.
• Variants that adjust scaling, initialization or the way the update is decomposed, such as DoRA and rsLoRA.

The PEFT library gathered these methods behind one interface so that users did not need to port code between research repositories. It integrates with Transformers and Accelerate, which is why a wrapped model can be passed to a standard trainer.

How it works

The workflow has five stages. Each maps to a few lines of code.

Step 1: Load the base model

Load a pretrained model as usual. Optionally load it in lower precision or in 4-bit form to save memory. The base weights will stay frozen.

Step 2: Choose a method and configure it

Create a config object that names the method and sets its options. For LoRA, the main options are the rank, the scaling factor, the dropout and the list of modules to adapt. For prompt-based methods, the main option is the number of virtual tokens.

Step 3: Wrap the model

Call a function that returns a PEFT model. Internally, the library finds the target layers, replaces or augments them with adapter-aware versions, freezes the base parameters and leaves the adapter parameters trainable.

Step 4: Train as usual

The wrapped model works with the normal training loop or a trainer. Only the adapter weights get gradients. The optimizer holds state only for them, which is where most memory is saved.

Step 5: Save, load, merge or swap

Saving writes only the adapter weights and a small config file. Loading attaches them to a fresh copy of the base model. You may also merge the adapter into the base weights to produce a standalone model with no extra run-time cost.

How LoRA works inside

For a weight matrix W of size d by k, LoRA keeps W frozen and learns two small matrices, A (r by k) and B (d by r), where r is much smaller than d and k. The layer output becomes the original output plus a scaled product of B, A and the input. Because r is small, the number of new parameters is r times (d plus k), far less than d times k.

Analogy: instead of redrawing a whole map, you add a thin transparent overlay that adjusts a few roads. The overlay is small, yet it changes the routes you take.

A numeric feel: for a 4096 by 4096 matrix, a full update is about 16.8 million numbers. With rank 8, LoRA has 8 times (4096 plus 4096), about 65 thousand numbers. That is under 0.4 percent of the full count for that matrix.

Which layers to adapt

Early use adapted only the query and value projections in attention. Many current recipes adapt all linear layers, including the feed-forward projections, which tends to give better quality at a modest memory cost. The choice is set by the list of target modules in the config.

Merging

Since the LoRA update is a matrix product that has the same shape as the original weight, you can add it into the weight once training ends. After merging, the model is a normal model and inference has no added latency. If you want to keep adapters separate, you keep a small extra computation per layer, in exchange for the ability to swap adapters on the fly.

Prompt-based methods inside

Prompt tuning learns a short sequence of vectors that is placed in front of the input embeddings. Prefix tuning learns vectors that are placed in the attention keys and values at each layer. Both leave the model weights untouched. They are very small, but they often work best with larger base models and can be harder to train than low-rank adapters.

Key concepts and terms

Rank and alpha, in plain words

Rank is the width of the adapter: a bigger rank can express more change. Alpha is a volume knob on the adapter output. The ratio of alpha to rank decides how strongly the adapter speaks compared with the base. Many people set alpha near twice the rank as a starting habit. Some variants change the scaling rule so that larger ranks stay stable, which is the idea behind rsLoRA.

A worked example

The following code fine-tunes a small causal language model with LoRA using PEFT and a standard trainer. Model and file names are placeholders.

Setup

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType

name = "example-org/example-1b"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()

The last line prints counts such as the number of trainable parameters, the total and the percentage. Expect a small percentage, often well below one percent for modest ranks.

Train with a trainer

from datasets import load_dataset
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling

data = load_dataset("json", data_files="train.jsonl")["train"]

def tokenize(ex):
    return tok(ex["text"], truncation=True, max_length=1024)

tokenized = data.map(tokenize, remove_columns=data.column_names)

args = TrainingArguments(
    output_dir="./outputs/lora-demo",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=2,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
)

trainer = Trainer(
    model=peft_model,
    args=args,
    train_dataset=tokenized,
    data_collator=DataCollatorForLanguageModeling(tok, mlm=False),
)
trainer.train()
peft_model.save_pretrained("./outputs/lora-demo/adapter")

Load the adapter later

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(name)
model = PeftModel.from_pretrained(base, "./outputs/lora-demo/adapter")
model.eval()

Merge for deployment

merged = model.merge_and_unload()
merged.save_pretrained("./outputs/lora-demo/merged")
tok.save_pretrained("./outputs/lora-demo/merged")

When the base was loaded in 4-bit form for training, merge into a copy of the base loaded in 16-bit form if quality matters, since merging into quantized weights can lose precision.

QLoRA variant

import torch
from transformers import BitsAndBytesConfig
from peft import prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb)
model = prepare_model_for_kbit_training(model)
peft_model = get_peft_model(model, config)

This is the recipe behind QLoRA: a frozen 4-bit base, LoRA adapters in higher precision, and an optimizer that pages state to host memory when needed. See bitsandbytes for the low-bit kernels.

Several adapters on one base

model.load_adapter("./adapters/support-tone", adapter_name="support")
model.load_adapter("./adapters/legal-style", adapter_name="legal")
model.set_adapter("support")   # use one
# later
model.set_adapter("legal")     # switch to another

This pattern lets one base model serve many tasks. Each adapter is small, so many fit in storage and memory.

A classification head example

For a sequence classification task, set the task type and list the new head under modules to save so it is trained in full and stored with the adapter.

config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8, lora_alpha=16, lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],
    modules_to_save=["score"],
)

A numeric walk-through

Say the base model has 1.0 billion parameters. With rank 16 on all linear layers across 24 layers, the adapter might hold around 10 to 20 million parameters, so roughly 1 to 2 percent. In 16-bit form, a 15 million parameter adapter is about 30 MB. Ten task adapters take about 300 MB, compared with 20 GB for ten full copies of the base model. That is the storage win.

For memory in training, full fine-tuning with a common optimizer needs the weights plus gradients plus two optimizer states per weight. Adapter training needs those extra states only for the adapter, so the dominant cost becomes the frozen weights and the activations. Gradient checkpointing, see gradient checkpointing, can cut activation memory further.

A debugging and sizing guide

Sizing the adapter before training

Use a quick formula. For each adapted linear layer with input size k and output size d, the adapter has r times (k plus d) parameters. Multiply by the number of adapted layers. A rank of 16 on seven projections in each of 32 layers gives a total in the tens of millions for a model of several billion parameters. Check the figure against the printed trainable count. If the two differ a lot, the target list is not what you think.

A step-by-step sanity routine

1. Wrap the model and print the trainable parameter count.
2. Run one forward pass before training. Because LoRA starts with the up matrix at zero, the output should equal the base model output.
3. Train for ten steps on a tiny batch. Loss should move.
4. Save the adapter and reload it into a fresh base. Outputs should match the trained model.
5. Merge, save and reload. Outputs should match again within small numeric noise.
6. Compare adapter-on and adapter-off outputs on a few prompts to confirm the adapter does something.

Reading symptoms

Versioning adapters

Treat an adapter like a software release. Give it a version, store its metrics, and record the base model identifier and revision. When the base model changes, retrain and re-evaluate rather than assuming the old adapter still fits.

Strengths and limitations

Strengths

• Low memory use during training, since optimizer state covers only the adapter.
• Small artifacts. Adapters are easy to store, share and version.
• Multi-task serving from one base model by swapping adapters.
• Less forgetting than full fine-tuning in many cases, since base weights are untouched.
• Wide method coverage under one interface.
• Works with quantized bases for single-GPU training of large models.
• Plays well with Transformers, Accelerate and common trainers.

Limitations

• Capacity. Very large domain shifts or new skills may need higher ranks or full fine-tuning.
• Hyperparameter sensitivity. Rank, alpha, target modules and learning rate all matter.
• Merge precision. Merging into low-bit weights can change outputs slightly.
• Extra latency if adapters stay separate and are not fused.
• Version drift. Adapters are tied to the base model they were trained on. A different base needs a new adapter.
• Not all architectures are supported out of the box. Custom layers may need manual target names.
• Prompt-based methods can be unstable on small models.
• A small file does not mean a small effect. Adapters can change behavior a lot, so evaluate them as seriously as a full model.

When quality lags full fine-tuning

Studies of low-rank methods often find that they match full fine-tuning on many tasks but lag on tasks that need learning many new facts or skills. If your results plateau, try a higher rank, adapt more layers, raise data quality, or compare with a short full fine-tuning run on a subset to see whether capacity is the bottleneck.

How it compares with alternatives

PEFT versus the libraries around it

PEFT versus retrieval

Fine-tuning changes how a model behaves: tone, format, reasoning habits and domain vocabulary. Retrieval changes what the model can look up: facts that change often. They solve different problems and are often combined.

Role in tabular foundation models

PEFT was designed for language models, yet its ideas carry over to models that learn from raw tables, because any pretrained transformer can receive adapters.

Background: how tabular foundation models work

A tabular foundation model such as TabPFN, TabICL or TabSTAR is pretrained on a very large number of tables, many of them synthetic. Each table has rows and columns of mixed types, may have missing values and may lack meaningful column names. The model reads labeled rows as context and predicts labels for new rows in one forward pass, which is in-context learning. Its attention often runs across both rows and columns, as in axial attention. The details are in /model/architecture.

Where PEFT helps

1. Domain specialization without full retraining. A pretrained tabular model can be adapted to a family of tables, such as sensor logs or transaction records, by training low-rank adapters on real tables from that domain. The base stays frozen, which limits damage to its general skills.
2. Continual learning. New data arrives over time. Instead of retraining the whole model, you can train a new adapter per period or per domain. This reduces forgetting, because earlier adapters and the base are untouched. Regularization methods such as EWC and replay methods such as experience replay can be combined with adapters. The broader topic is covered in /model/research.
3. Semantic and sector heads. A small head on top of table embeddings can predict column meaning or record sector. PEFT can train such a head together with low-rank updates in the encoder. See semantic sector heads.
4. Imputation and cell modeling. A model pretrained with masked-cell objectives can be tuned on a company's tables to fill missing values better. Adapters keep this cheap. See masked cell modeling and imputation networks.
5. Anomaly detection. Adapters can specialize an encoder to a normal operating regime, so reconstruction or likelihood scores become more sensitive to deviations in that regime. See anomaly detection.
6. Language side of table work. When a language model reads serialized rows, labels columns or explains outliers, PEFT with LoRA is a standard way to tune it.

Practical notes for table models

• Attention layers in table models may use different names than language models. Print the module names and choose target modules by inspection.
• Numeric embeddings and type embeddings may need to be trained in full, so list them under modules to save.
• Because in-context learning already adapts a model at inference time without gradients, fine-tuning is worth it mainly when the target tables differ sharply from the pretraining distribution, or when the context length limit forces you to compress many rows into learned knowledge.
• Evaluate on held-out tables, not just held-out rows. Table-level shift is the real test. See /model/benchmarks.

Honest limits

Support depends on the architecture and the version of the library. Not every tabular model exposes standard linear layers that PEFT can find automatically. Always check the trainable parameter count after wrapping, and run a quick sanity test comparing outputs before and after wrapping with a zero-initialized adapter.

Practical guidance and best practices

Start from a known recipe

A good first LoRA recipe: rank 16, alpha 32, dropout 0.05, all linear layers, learning rate around 1e-4 to 2e-4, bf16, cosine schedule with short warmup. Adjust from there based on validation results.

Print what is trainable

After wrapping, print trainable parameter counts. If the count is zero or equals the full model, something is wrong.

Choose target modules by inspection

List the model's module names and pick the linear layers you want. Wrong names silently adapt nothing in some setups, or raise an error in others.

for n, m in model.named_modules():
    if m.__class__.__name__ == "Linear":
        print(n)

Match the learning rate to the method

These are habits, not laws. Run a small sweep.

Save more than weights

Store the adapter, its config, the base model name and revision, the tokenizer version, the dataset version and the training config. An adapter without its base model name is a puzzle.

Evaluate before and after

Compare the tuned model with the base on a held-out set that matches the real task. Also test general abilities to detect forgetting.

Merge with care

If you merge, test the merged model against the unmerged one on a few prompts. Outputs should be close. If you merge into a quantized model, expect small differences.

Manage many adapters

Name adapters clearly, keep a registry that links each to its base, data and metrics, and test loading in a clean environment.

Reduce memory further

• Enable gradient checkpointing.
• Use bf16 or fp16, see mixed precision.
• Use a quantized base.
• Lower sequence length or micro batch size and raise accumulation steps.
• Shard the base across GPUs with FSDP or DeepSpeed ZeRO when the base is large.

A short decision guide

Common mistakes

• Wrong target module names. The adapter attaches to nothing, or to the wrong layers.
• Forgetting the head. A new classification head stays random because it was not listed as trainable.
• Rank too small for the task. Training loss plateaus high.
• Rank too large for the data. The adapter overfits.
• Learning rate copied from full fine-tuning. Adapters often need a higher rate, and a too-low rate looks like nothing is learning.
• Loading an adapter on the wrong base. Shapes may match while behavior is garbage.
• Merging into a quantized base without testing.
• Mixing tokenizer versions. Token ids shift and output degrades.
• Training with gradient checkpointing but without enabling input gradients. Some setups need a call that makes input embeddings require gradients, otherwise gradients vanish.
• Saving only part of the state. Extra trainable modules are lost.
• Not freezing what you think is frozen. Check parameter flags.
• Judging by loss alone. Compare outputs on real tasks.
• Treating adapters as risk-free. They can change safety behavior. Re-run safety tests after tuning.

Frequently asked questions

What is PEFT used for?

PEFT is used to fine-tune large pretrained models at low cost by training a small number of added parameters instead of all weights. It provides methods such as LoRA, prefix tuning, prompt tuning and IA3, and it integrates with the usual Transformers training workflow.

What is the difference between PEFT and LoRA?

PEFT is a family of techniques and a library, while LoRA is one specific technique inside it. LoRA adds low-rank update matrices to chosen layers. The PEFT library implements LoRA along with several other methods behind a common interface.

How much memory does PEFT save?

It saves most of the optimizer and gradient memory, because only the adapter weights are trained. Combined with a 4-bit base as in QLoRA, models with billions of parameters can often be tuned on a single GPU. The exact saving depends on model size, sequence length and settings.

Can I merge a PEFT adapter into the base model?

Yes. For LoRA, a merge step adds the low-rank update into the base weights and gives a standalone model with no extra latency. It is wise to test the merged model against the unmerged one, and to merge into a full-precision copy if the base was quantized during training.

Can I use several adapters on one base model?

Yes. The library supports loading multiple adapters with different names and switching between them. This lets one base model serve many tasks, with each adapter stored as a small file.

How do I choose the LoRA rank?

Start with a moderate value such as 16, then test a lower and a higher rank on your own validation set. Small ranks suit style and format changes, while larger ranks suit broader domain shifts. Beyond a point, more rank adds cost without gains.

Does PEFT work with quantized models?

Yes. PEFT works with bitsandbytes 4-bit and 8-bit loading, which is the basis of QLoRA. The base weights are stored in low-bit form and frozen, while adapter weights are kept in higher precision and trained.

Is parameter-efficient tuning as good as full fine-tuning?

On many tasks it comes close, especially for style, format and moderate domain adaptation. On tasks that require learning a lot of new knowledge, full fine-tuning may do better. Compare on your own task, using a held-out test set.

Can PEFT be used for tabular or table-based models?

It can be used with any transformer whose layers PEFT can target. For table models, adapters can specialize a pretrained model to a domain, support continual learning and tune semantic or sector heads. Check module names and list any numeric or type embeddings that must be trained in full.

Do adapters work across different base models?

No. An adapter is trained for a specific base model and architecture. Loading it onto a different base gives wrong or broken results even if shapes happen to match. Keep a record of the exact base each adapter belongs to.

Glossary

• Adapter: A small trainable module attached to a frozen model.
• Alpha: A scaling factor for the LoRA update.
• Base model: The pretrained model that stays frozen.
• Bottleneck adapter: A small down and up projection placed between layers.
• DoRA: A LoRA variant that separates magnitude and direction of weight updates.
• Fine-tuning: Continued training on a narrower dataset.
• Frozen: Not updated during training.
• IA3: A method that learns scaling vectors for activations.
• LoRA: Low-rank adaptation of weight matrices.
• Merge: Folding adapter weights into base weights.
• Modules to save: Layers trained fully and saved with the adapter.
• NF4: A 4-bit data type used in QLoRA.
• Prefix tuning: Learning vectors placed in attention keys and values.
• Prompt tuning: Learning input embeddings prepended to the prompt.
• QLoRA: LoRA training on a 4-bit quantized base.
• Rank: The inner dimension of low-rank matrices.
• Target modules: Layers that receive adapters.
• Virtual tokens: Learned vectors acting like extra prompt tokens.

Summary

PEFT is the library that makes parameter-efficient fine-tuning practical. It freezes a base model and trains small add-ons such as low-rank matrices, virtual tokens or scaling vectors. This cuts training memory, produces tiny artifacts, allows one base model to serve many tasks and often limits forgetting. Its limits are capacity on large domain shifts, sensitivity to rank and learning rate, base-model coupling and care needed when merging.

Good practice is to start from a known recipe, inspect module names, print trainable counts, save adapters with their metadata, evaluate against the base and test merged models. In table work, the same adapter idea supports domain specialization, continual learning, semantic and sector heads, imputation and anomaly tuning for pretrained tabular models, as long as the architecture exposes layers that can be targeted. Related pages include LoRA, QLoRA, TRL, Axolotl, bitsandbytes, /model/architecture and /model/benchmarks.

Related

It belongs to Training tooling in my stack, under LLMs, retrieval, and evaluation. Others in the same group: