LoRA

Overview

LoRA, short for low-rank adaptation, is a way to fine-tune a large neural network by training only a tiny set of new parameters. The original weights stay frozen. For each chosen weight matrix, LoRA learns two small matrices whose product is a low-rank update. That update is added to the frozen weight at run time. Often well under one percent of the parameters are trained, yet quality on the target task can come close to full fine-tuning.

The practical payoff is large. Memory for gradients and optimizer states shrinks because they exist only for the small matrices. The saved file is tiny, often a few megabytes to a few hundred megabytes. You can keep many adapters over a single base model and swap them per task or per customer. After training, the update can be merged into the base weights, so inference costs nothing extra. LoRA is the default member of the PEFT family and the starting point for variants such as QLoRA, DoRA, and rsLoRA.

What it is

LoRA is a parameter-efficient fine-tuning method. It does not change the model architecture in a way that survives training, because the learned update can be folded back into the original weights. During training it adds a parallel low-rank path next to selected linear layers.

The core idea in one sentence

Instead of learning a full update matrix with millions of numbers, learn a thin product of two small matrices that approximates the update.

An everyday analogy

Think of a large printed atlas. You want it to reflect a new city district. Reprinting the whole atlas is full fine-tuning. LoRA is a transparent overlay sheet that you lay over one page. The overlay is small and cheap. You can keep many overlays for many districts. If you like one, you can print it permanently into the page, which is merging.

What gets adapted

• Linear projection layers in attention, usually the query and value projections, and often all four attention projections.
• Feed-forward linear layers, which many recent recipes include for better quality.
• Embedding layers and the output head in some setups, although these are often handled separately.
• Convolution layers, through a reshaped variant, in vision and diffusion models.

What stays frozen

• All original weights of the base model.
• Normalization parameters, unless you choose to unfreeze them.
• Biases, unless you choose to train them.

Why it is called low rank

The rank of a matrix is the number of independent directions it uses. A matrix of size 4096 by 4096 can have rank up to 4096. A product of a 4096 by 8 matrix and an 8 by 4096 matrix has rank at most 8. LoRA bets that the useful change needed for a new task lives in a small number of directions. See linear algebra for the underlying ideas.

Where it came from

LoRA belongs to a line of work on cheap adaptation of big pre-trained models.

Earlier steps

• Transfer learning showed that a model trained on a broad task can be adapted to a narrow one with little data. See transfer learning.
• Bottleneck adapters inserted small trainable modules between frozen layers, around 2019. They worked but added latency at inference because the modules stayed in the forward path.
• Prefix tuning and prompt tuning learned virtual tokens instead of weights. They are cheap but can be harder to optimize and use up context length.
• Studies of intrinsic dimensionality, around 2020, showed that fine-tuning large models can succeed in a surprisingly small subspace of the full parameter space. This is the key empirical fact that motivates LoRA.

The LoRA paper

The paper "LoRA: Low-Rank Adaptation of Large Language Models" appeared in 2021. It proposed freezing the pre-trained weights and injecting trainable rank decomposition matrices into the Transformer layers. It stressed three properties: far fewer trainable parameters, no added inference latency after merging, and easy task switching by swapping the small matrices.

What followed

• QLoRA, 2023, combined LoRA with a 4-bit quantized base model.
• rsLoRA, 2023, fixed the scaling factor so higher ranks train well. See rsLoRA.
• DoRA, 2024, split weights into magnitude and direction.
• GaLore, 2024, took a different route and applied low-rank ideas to gradients instead of weights.
• Many library integrations followed, including support in Hugging Face tooling, Unsloth, Axolotl, torchtune, and LLaMA-Factory.

How it works

Step 1: pick the layers

You choose which linear layers receive adapters. In library terms these are called target modules. A common first choice is the attention query and value projections. A stronger choice is every linear layer in the block.

Step 2: freeze the base

Set the requires-gradient flag to false for all original parameters. Autograd then skips computing gradients for them. This is where most of the memory saving comes from.

Step 3: add the low-rank path

For a frozen weight matrix W with shape d_out by d_in, LoRA adds two matrices:

• A, with shape r by d_in. It projects the input down to r dimensions.
• B, with shape d_out by r. It projects back up to the output size.

The rank r is small, for example 4, 8, 16, 32, or 64.

Step 4: combine the paths

The layer output becomes:

y = W x + (alpha / r) * B (A x)

The first term is the frozen path. The second term is the learned correction. The factor alpha over r is a scaling constant that controls how strongly the update speaks.

Step 5: initialize safely

• A is initialized with small random values, usually from a Gaussian or uniform distribution.
• B is initialized to zeros.

Because B is zero, the product B A is zero at the start. The model begins exactly as the pre-trained model and drifts only as training moves B away from zero. This keeps early training stable.

Step 6: train only A and B

The optimizer sees only the adapter parameters. Optimizer states such as the two moment estimates in Adam exist only for them. For a seven billion parameter model with a few tens of millions of adapter parameters, this removes the bulk of optimizer memory.

Step 7: deploy

You have three choices.

• Keep the adapter separate and load it on top of the base model. This is flexible for multi-tenant serving.
• Merge the adapter into W with W' = W + (alpha / r) * B A. The model is then a plain model with no added latency.
• Serve many adapters over one base with a runtime that batches requests across adapters.

Parameter counting

For one layer with d_in and d_out, full fine-tuning trains d_in times d_out numbers. LoRA trains r times (d_in + d_out). For a square layer of size 4096 and rank 8, full training touches about 16.8 million numbers while LoRA touches 65,536. That is a ratio of roughly 256 to one for that layer.

An analogy for rank

Imagine mixing paint. A full update lets you adjust every pigment independently in a million tiny regions. A low-rank update says: the change you need is a blend of just a few base colors spread over the picture. If the new task is a small stylistic shift, a few base colors are enough.

Why the scaling factor matters

The ratio alpha over r multiplies the adapter output. If you change r but keep alpha fixed, the effective step size changes. Many practitioners set alpha to a multiple of r, such as alpha equal to r or twice r, to keep behavior similar when exploring ranks. The rsLoRA article explains why dividing by the square root of r is better at high rank.

Where the memory goes

Activations are not reduced by LoRA, so gradient checkpointing and mixed precision are still useful companions.

Key concepts and terms

A worked example

A numeric walk-through

Take a tiny layer with d_in equal to 4, d_out equal to 3, and rank 1.

• W is a 3 by 4 frozen matrix.
• A is 1 by 4, for example [0.1, -0.2, 0.3, 0.0].
• B is 3 by 1, initialized to [0, 0, 0].

At step zero, B A is a 3 by 4 matrix of zeros, so the output equals W x. After a few updates, B might become [0.5, -0.1, 0.2]. Then B A is a 3 by 4 matrix where every row is a multiple of A. This is a rank-one update: all rows point in the same direction, scaled by the entries of B. Only 7 numbers (4 plus 3) were learned, versus 12 for a full update. The saving grows quickly with size.

A code example with the PEFT library

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_name = "example-org/base-model"  # placeholder name
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16
)

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
model.print_trainable_parameters()
# prints something like: trainable params: tens of millions || all params: billions

Training loop sketch

from transformers import Trainer, TrainingArguments

args = TrainingArguments(
    output_dir="out/adapter-a",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    num_train_epochs=2,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=20,
    save_strategy="epoch",
)

trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train()
model.save_pretrained("out/adapter-a")  # saves only the adapter

Merging for deployment

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
tuned = PeftModel.from_pretrained(base, "out/adapter-a")
merged = tuned.merge_and_unload()
merged.save_pretrained("out/merged-model")

A from-scratch layer

This shows the mechanism without a library.

import math
import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r: int = 8, alpha: int = 16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

Reading the result

After training you compare against the base model and against full fine-tuning on a held-out set. If LoRA lags, raise the rank, add more target modules, or train longer before concluding that the method is the limit.

Strengths and limitations

Strengths

• Very few trainable parameters, which cuts gradient and optimizer memory.
• Small files that are easy to store, version, and share.
• No inference latency after merging.
• Easy task switching and per-tenant customization.
• Works with most Transformer-style models and many other architectures.
• Frozen base reduces the risk of damaging general knowledge, a milder form of forgetting.
• Composes with quantization, as in QLoRA.
• Well supported across training and serving tools.

Limitations

• Capacity is bounded by the rank. Tasks that need large changes to the model can fall short.
• Learning new facts or a new language from scratch is harder than shifting style or format.
• Activation memory is unchanged, so long sequences still need care.
• Hyperparameters interact: rank, alpha, learning rate, and target modules.
• Merging adapters trained on different tasks does not always combine cleanly.
• If the base is quantized, merging into the low-bit weights needs extra care.
• Standard LoRA scaling can slow learning at high rank, which rsLoRA addresses.
• It may trail full fine-tuning on hard tasks, which motivated DoRA.

When the gap to full fine-tuning is small

• Instruction following and chat style.
• Classification and extraction with moderate data.
• Domain tone and format adaptation.
• Small tasks with a few thousand examples.

When the gap can be larger

• Heavy domain shift, such as new scripts or specialist notation.
• Large data volumes where a higher capacity is useful.
• Tasks that require precise numerical reasoning learned from scratch.

How it compares with alternatives

A rule of thumb

If you have enough memory, full fine-tuning gives the highest ceiling. If you do not, LoRA is the first thing to try. If the base does not fit in memory at all, move to QLoRA. If optimizer memory is the issue even with full updates, consider GaLore.

Role in tabular foundation models

A tabular foundation model learns from raw tables. Its inputs have a row and column structure, mixed types such as numbers, categories, dates and text, missing cells, and often no useful column names. See /model/architecture for one concrete design. LoRA matters for these models in several concrete ways.

Adapting a pre-trained table model to a new schema

Models such as TabPFN, TabICL, TabSTAR, and FT-Transformer are pre-trained on many tables, often synthetic ones. A new customer table has its own column mix, value ranges, and quirks. Full fine-tuning of the whole model per customer is costly and risks forgetting. A LoRA adapter per table family or per customer gives a small, separate artifact over one shared base.

In-context learning and adapters are complementary

Many tabular foundation models solve a task by reading labeled rows in the context and predicting new rows without gradient updates. That is in-context learning. LoRA is a gradient-based route. The two can be combined. In-context learning handles the quick case. A LoRA adapter captures systematic shifts that context alone cannot fix, such as an unusual feature distribution, heavy skew, or a domain-specific link between columns.

Mixed types and missing values

Tabular encoders often have separate input paths for numeric, categorical, and text columns, plus a representation for missing cells. LoRA can target the projection layers on those paths. This lets the model learn how a particular schema encodes missingness, for example a sentinel value that really means not applicable, without retraining the core table reasoning.

No column names

When columns have no names, the model relies on value statistics and relationships between columns. A short adapter trained on a few tables from a sector can teach the model sector-specific regularities, such as typical ranges and co-occurrence patterns, while the frozen base keeps its general table skills.

Continual learning

Continual learning means learning new tasks over time without losing old ones. Because the base is frozen, each new task gets its own adapter. Old tasks remain intact since their adapters are untouched. This is a clean, practical defence against catastrophic forgetting, related to ideas such as EWC but simpler to operate. Routing then chooses which adapter to load.

Imputation

Imputation fills in missing cells. A foundation model can predict masked cells directly, a task closely tied to masked cell modeling and imputation networks. LoRA can specialize the imputer to a table where missingness is not random, for example values missing because a sensor was off. Training on masked versions of the target table with a LoRA adapter is cheap and keeps the base intact.

Anomaly detection

For anomaly detection, a model can learn what normal rows look like and score departures. A LoRA adapter can adapt the notion of normal to a specific system, such as one plant line or one billing pipeline, using only normal data. Separate adapters per asset type avoid mixing different notions of normal.

Sector and semantic classification

Tasks such as assigning a table or column to a sector or semantic type use classification heads on top of table embeddings. See semantic sector heads. LoRA lets you tune the encoder lightly toward a new label set or a new industry vocabulary, and a new head is trained alongside it.

Practical fit

Honest caveats

• Table models are often much smaller than language models, so the memory savings are smaller in absolute terms. The operational benefits of small artifacts and no forgetting remain.
• Not every tabular architecture has the same linear layers as a language model, so target module choice must be checked per model.
• For in-context models, check whether a few extra context rows solve the problem before training anything.
• Tune on a held-out table, not only on held-out rows, so that you measure generalization to new schemas. See /model/benchmarks for how tabular models are evaluated and /model/research for background.

Practical guidance and best practices

Choosing the rank

• Start with rank 8 or 16 for style and format tasks.
• Try 32 or 64 for harder tasks or larger data.
• Increase the rank if training loss plateaus high while the base fits the task type.
• Do not assume more is better: with standard scaling, high rank may not help. See rsLoRA.

Choosing alpha

• A common default is alpha equal to r, or two times r.
• Keep the ratio constant when you compare ranks, or the comparison is unfair.
• If training diverges, lower alpha or the learning rate.

Choosing target modules

• Attention only is the smallest option.
• All linear layers is the common modern default and usually gives better results.
• Check layer names for your model, because they differ between architectures.

Learning rate

• LoRA tolerates and often needs a higher learning rate than full fine-tuning. Values around ten times higher are typical starting points.
• Use warmup and a decaying schedule.
• Watch the loss curve in the first hundred steps.

Data

• Quality beats quantity. A clean set of a few thousand examples can be enough for format tasks.
• Hold out a validation set that reflects the real target distribution.
• Keep the prompt or input format identical between training and use.

Memory tips

• Use bf16 where the hardware supports it. See mixed precision.
• Turn on gradient checkpointing for long sequences.
• Use gradient accumulation to reach a useful effective batch size.
• Combine with 4-bit loading through bitsandbytes when the base is too large. See QLoRA.
• For multi-GPU runs, consider FSDP or DeepSpeed ZeRO.

Evaluation

• Compare to the base model with the same prompts.
• Compare to a stronger baseline such as full fine-tuning on a small budget.
• Check for regressions on general tasks.
• Log rank, alpha, modules, seed, and data version so runs can be reproduced.

Serving

• Merge for a single fixed task.
• Keep adapters separate for multi-tenant serving, and use a runtime that supports loading several. See vLLM.
• Version adapters together with the exact base model checkpoint they were trained on.

Governance

• An adapter is only valid for its base. Store the base identifier in the adapter metadata.
• Treat adapters like code artifacts: review, test, and sign them.

Common mistakes

• Setting B to nonzero at start. This breaks the guarantee that training begins at the base model.
• Using the same learning rate as full fine-tuning. It is often too low for LoRA.
• Changing rank without adjusting alpha, then blaming the rank for the change in behavior.
• Targeting too few modules and concluding that LoRA cannot learn the task.
• Merging into a quantized base without a plan, which can lose precision.
• Loading an adapter on a different base version than it was trained on.
• Evaluating only on training-like prompts, which hides overfitting.
• Training for too many epochs on a small set. LoRA can still memorize.
• Forgetting that padding and masking must match training in inference.
• Assuming adapters from different tasks can always be added together.
• Ignoring activation memory and running out of memory at long sequence length.
• Comparing LoRA to full fine-tuning with a poorly tuned LoRA learning rate.
• Treating the adapter file as self-contained when it needs the exact tokenizer and chat template.

Frequently asked questions

What does LoRA stand for and what does it do?

LoRA stands for low-rank adaptation. It fine-tunes a large model by keeping the original weights frozen and training two small matrices per chosen layer, whose product forms a low-rank update added to the layer output. This makes training cheap in memory and produces small adapter files.

How many parameters does LoRA train?

It trains r times the sum of the input and output sizes for each adapted layer. In practice this is often well under one percent of the model, and it depends on the rank and on how many layers you adapt. Higher rank and more target modules raise the count.

Does LoRA slow down inference?

Not after merging. If you add the scaled product of the two matrices into the original weight, the model is a normal model again. If you keep the adapter separate, there is a small extra computation per adapted layer, which is the price of easy swapping.

What rank should I use?

Start with 8 or 16 for simple tasks and move to 32 or 64 when the task is harder or the dataset is large. If you raise the rank, keep the ratio of alpha to rank sensible and check that loss actually improves. Evidence beats rules of thumb here.

Is LoRA as good as full fine-tuning?

On many tasks it is close, especially for style, format, and moderate domain shifts. On tasks that need large changes, it can trail. Using all linear layers, a good learning rate, and a suitable rank narrows the gap, and variants such as DoRA aim to close it further.

Can I train several LoRA adapters and use them together?

Sometimes. You can load several adapters and switch between them, which is reliable. Adding or averaging adapter updates can work when the tasks are related, but it is not guaranteed, so test the combined model on each task before relying on it.

How is LoRA different from QLoRA?

LoRA describes the adapter method. QLoRA runs the same adapters on top of a base model stored in 4-bit precision, which lowers memory enough to fine-tune very large models on a single GPU. The adapters themselves are still trained in higher precision.

Can LoRA be used outside language models?

Yes. It applies to any model with linear or convolution layers, including vision Transformers, diffusion models, speech models, and tabular models. The same ideas hold: freeze the base, learn a small low-rank update, and merge or swap as needed.

Does LoRA prevent catastrophic forgetting?

It reduces it, because the base weights never change and the adapter can be removed. The adapted model can still drift on tasks outside the adapter training data while the adapter is active. Using one adapter per task and routing between them is the safer pattern.

When should I choose something other than LoRA?

Choose full fine-tuning when you have the memory and need the highest ceiling, QLoRA when the base does not fit, and GaLore when optimizer memory limits full-weight training. Prompt-style methods can be enough for very light adaptation of very large models.

Glossary

• Adapter: A small trainable module or set of matrices added to a frozen model.
• Alpha: The numerator of the LoRA scaling factor.
• Autograd: The system that computes gradients automatically.
• Base model: The pre-trained model that stays frozen.
• Catastrophic forgetting: Loss of earlier skills after training on new data.
• Fine-tuning: Further training of a pre-trained model on a narrower task.
• Gradient: The direction and size of change that reduces the loss.
• In-context learning: Solving a task from examples placed in the input, with no weight updates.
• Low-rank: Described by a small number of independent directions.
• Merge: Folding the adapter update into the base weights.
• Optimizer state: Extra values such as moment estimates that an optimizer stores per parameter.
• Rank: The inner dimension of the adapter matrices.
• Target modules: The layers that receive adapters.
• Trainable parameters: The parameters that receive gradient updates.

Summary

LoRA freezes a pre-trained model and learns a small low-rank correction for selected layers. That correction is the product of two thin matrices, scaled by alpha over rank, and it starts at zero so training begins from the base model. The method cuts gradient and optimizer memory, creates tiny portable adapters, and adds no inference cost once merged. Its limits are bounded capacity, sensitivity to rank, alpha, and learning rate, and unchanged activation memory.

For tabular foundation models, LoRA offers a practical way to adapt a shared pre-trained table model to new schemas, shifted distributions, imputation patterns, anomaly baselines, and sector labels, with one small adapter per task and no forgetting of earlier tasks. Related reading: QLoRA, DoRA, rsLoRA, GaLore, PEFT, and the research and benchmarks pages.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: