DoRA

Overview

DoRA, short for weight-decomposed low-rank adaptation, is a parameter-efficient fine-tuning method that builds on LoRA. It rewrites each pre-trained weight matrix as two parts: a magnitude and a direction. LoRA-style low-rank matrices update the direction, while a small trainable vector learns the magnitude. The idea is that this split lets learning behave more like full fine-tuning, so DoRA often narrows the quality gap that standard LoRA leaves.

DoRA keeps the main practical benefits of LoRA. The base stays frozen, only a small number of parameters train, and the update can be merged into the weights after training, so inference cost does not grow. The extras are a modest increase in trainable parameters and some extra compute during training. DoRA is supported in common libraries such as PEFT, and it combines with quantized bases as in QLoRA. It is best seen as a drop-in upgrade to LoRA for cases where extra quality is worth a small cost.

What it is

DoRA is a modification of LoRA that decomposes each weight into magnitude and direction, then adapts them separately.

The one-sentence version

Keep a trainable length for each weight column, keep a unit-length direction made from the frozen weight plus a low-rank update, and multiply them together.

An everyday analogy

Think of a set of arrows. Each arrow has a direction and a length. LoRA changes the arrows by adding small nudges, and the nudge changes both direction and length at once in a tangled way. DoRA says: let one small dial control how long each arrow is, and let the low-rank update steer only where each arrow points. Two simple dials are easier to turn than one tangled one.

What is different from LoRA

What stays the same

• The base model is frozen.
• The low-rank matrices A and B are trained with the same shapes as in LoRA.
• Hyperparameters such as rank, alpha, and target modules keep their meaning.
• Adapter files stay small.

Where it came from

Weight normalization

The idea of splitting a weight into a length and a direction is older than DoRA. Weight normalization, proposed in the mid 2010s, reparameterized each weight vector as a scalar magnitude times a unit direction to make optimization easier. The scalar and the direction can then be learned with separate dynamics.

The motivation from analyzing fine-tuning

The DoRA work, published in 2024, analyzed how full fine-tuning and LoRA change weights. It looked at how magnitude and direction move during training. Full fine-tuning showed a certain relationship between changes in the two, while LoRA showed a different, more tightly coupled pattern. The authors argued that this coupling limits LoRA's ability to mimic full fine-tuning, and that decoupling the two should help.

The paper

The paper is titled "DoRA: Weight-Decomposed Low-Rank Adaptation." It proposed decomposing the pre-trained weight into magnitude and direction, using LoRA only for the directional update, and training the magnitude directly. It reported gains over LoRA across language and vision-language tasks at similar parameter counts, according to its experiments.

After the paper

• Support was added to widely used libraries, so DoRA is often a configuration flag on a LoRA setup.
• Faster implementations reduced the training overhead by avoiding repeated recomputation of norms.
• Combinations with quantized bases, with larger ranks, and with other LoRA variants such as rsLoRA were explored.

How it works

Step 1: write a weight as magnitude and direction

Take a frozen weight matrix W with shape d_out by d_in. For each column, compute its length. The length of a column is the square root of the sum of squares of its entries. Dividing the column by its length gives a unit direction. In the original formulation, the norm is taken per column, so the magnitude is a vector with one number per column.

W = m * (W / ||W||_c)

Here m is the magnitude vector and the second factor is the matrix with each column normalized. At the start, m equals the column norms of W, so the decomposition reproduces W exactly.

Step 2: add the low-rank update to the direction

As in LoRA, learn A and B with a small inner rank r. Form the updated matrix:

V = W + (alpha / r) * B A

Step 3: normalize the updated direction

Divide V by its column norms to get a unit direction again.

direction = V / ||V||_c

Step 4: apply the learned magnitude

Multiply by the trainable magnitude vector m:

W' = m * direction

The layer then computes y = W' x. Only m, A, and B are trained.

Step 5: initialization

• B starts at zero, so V equals W at the start.
• A starts small and random, as in LoRA.
• m starts as the column norms of W.

So W' equals W at step zero, and the model starts as the base model.

Step 6: backpropagation and the normalization term

Gradients must flow through the normalization step. Computing norms of the full matrix V in every step costs extra compute and memory. Implementations treat the norm as a constant when computing some gradients to reduce this cost, a practical detail that keeps the overhead moderate.

Step 7: merge for inference

After training, compute W' once and replace the original weight. The merged model is a plain model with no extra parameters and no added latency. This holds for DoRA just as it does for LoRA.

Why separating magnitude helps

Imagine adjusting a speaker. One knob is volume, the other is tone balance. If a single knob changes both in a coupled way, it is hard to reach a target setting with few adjustments. Two separate knobs let you reach it quickly. In DoRA, the magnitude vector is the volume knob and the low-rank update is the tone control.

What the optimizer sees

In plain LoRA, the gradient for A and B has to push the weight both toward a new direction and toward a new length. These two jobs compete, and the low-rank update may need extra capacity to do both. In DoRA, the length has its own parameter with its own gradient. The low-rank matrices can focus on direction. Think of a car with separate pedals for speed and steering, instead of one lever that does both.

A note on the normalization axis

The decomposition can be taken along columns or rows, and the choice depends on how a framework stores the weight. What matters is that each output feature or each input feature gets one learned length. Libraries pick the convention that matches the layer layout, so you rarely set it by hand. If you write custom code, test that the merged weight at initialization equals the original weight to numerical precision.

Parameter count

For a layer with d_in and d_out, DoRA trains r times (d_in + d_out) for the adapter plus d_out or d_in for the magnitude vector, depending on the normalization axis. For large layers the magnitude vector is tiny compared with the low-rank matrices, so DoRA adds only a small increase over LoRA.

Key concepts and terms

A worked example

A tiny numeric walk-through

Take a 2 by 2 frozen weight W with columns c1 = [3, 4] and c2 = [0, 2].

1. Column norms: ||c1|| = 5 and ||c2|| = 2. So m = [5, 2].
2. Unit directions: d1 = [0.6, 0.8] and d2 = [0, 1].
3. Suppose the low-rank update adds [0.3, 0.0] to c1 and [0.0, 1.0] to c2. Then V has columns [3.3, 4.0] and [0, 3].
4. New norms: ||[3.3, 4.0]|| is about 5.19 and ||[0, 3]|| is 3.
5. New directions: [0.636, 0.771] and [0, 1].
6. Apply magnitude. Initially m = [5, 2], so W' has columns [3.18, 3.85] and [0, 2].

Note what happened to the second column. The low-rank update stretched it from length 2 to length 3, but after normalization its length is restored to the learned magnitude of 2. The length of each column is now controlled only by m. During training, m itself will move if the task wants longer or shorter columns. This is the decoupling at work.

A code example with PEFT

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained(
    "example-org/base-model", torch_dtype=torch.bfloat16  # placeholder
)

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    use_dora=True,          # turns on weight decomposition
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
model.print_trainable_parameters()

The only change from a LoRA setup is the DoRA flag. This is why DoRA is called a drop-in upgrade.

A minimal layer sketch

import torch
import torch.nn as nn
import torch.nn.functional as F

class DoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        out_f, in_f = base.weight.shape
        self.A = nn.Parameter(torch.randn(r, in_f) * 0.01)
        self.B = nn.Parameter(torch.zeros(out_f, r))
        self.scale = alpha / r
        # magnitude per output row (torch stores weight as out x in)
        self.m = nn.Parameter(base.weight.norm(p=2, dim=1).detach().clone())

    def forward(self, x):
        W = self.base.weight
        V = W + self.scale * (self.B @ self.A)
        norm = V.norm(p=2, dim=1, keepdim=True).detach()
        W_eff = self.m.unsqueeze(1) * (V / norm)
        return F.linear(x, W_eff, self.base.bias)

This sketch normalizes along each output row because the common framework stores a linear weight as output by input. Libraries handle these axis conventions for you.

Comparing LoRA and DoRA fairly

When you test DoRA against LoRA, change only one thing. Keep rank, alpha, data order, seed, learning rate schedule, and target modules the same. Run a few seeds, because differences between methods can be of the same size as seed noise. Report the mean and spread.

A merge example

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "example-org/base-model", torch_dtype=torch.bfloat16  # placeholder
)
tuned = PeftModel.from_pretrained(base, "out/dora-adapter")
merged = tuned.merge_and_unload()
merged.save_pretrained("out/dora-merged")

Strengths and limitations

Strengths

• Often closer to full fine-tuning quality than LoRA at similar trainable parameter counts.
• Can work well at lower ranks, which keeps adapters small.
• Same merge story as LoRA, so there is no inference overhead after merging.
• Simple to enable in common libraries.
• Compatible with quantized bases and with standard training tools.
• The decomposition has a clear intuitive meaning, which makes analysis of what training changed easier.

Limitations

• Extra compute and memory during training compared with LoRA, because of the normalization step.
• Slightly more trainable parameters than LoRA.
• Gains are task dependent. On easy tasks the difference may be within noise.
• Support in serving systems for unmerged DoRA adapters is less widespread than for LoRA.
• Hyperparameters still matter: a poorly tuned DoRA can lose to a well-tuned LoRA.
• On a quantized base, the normalization needs dequantized weights, which adds more overhead.

When DoRA is a good fit

• Quality matters more than a small increase in training time.
• You are already using LoRA and see a gap to full fine-tuning.
• You want good results at a small rank.
• The task involves changing behavior in subtle ways, such as style, reasoning format, or domain terms.

When plain LoRA is enough

• Quick experiments where speed matters most.
• Very easy tasks where LoRA already matches the target.
• Serving setups that depend on unmerged adapters with tooling built only for LoRA.

How it compares with alternatives

How DoRA relates to the other LoRA variants

• DoRA changes what is learned (magnitude and direction).
• rsLoRA changes how the update is scaled.
• QLoRA changes how the base is stored.
• These ideas address different issues, so they can be combined. Test combinations rather than assuming they add up.

Decision table

Choosing

Begin with LoRA. If the validation metric lags behind a full fine-tuning reference, or if you must use a low rank, switch on DoRA and compare. If you see no gain across seeds, keep LoRA for its speed.

Role in tabular foundation models

Tabular foundation models learn from raw tables, with rows and columns, mixed numeric, categorical, date, and text values, missing cells, and often no useful column names. They are pre-trained on large numbers of synthetic or real tables and solve tasks through in-context learning. See /model/architecture, /model/research, and /model/benchmarks.

Why a magnitude and direction split suits tables

Tables are numerically sensitive. A model that reads cell values must keep the scale of its internal features calibrated. When you adapt a table model to a new schema, two things often change: which patterns matter (direction) and how strongly each feature should count (magnitude). Standard LoRA mixes both in one update. DoRA gives the model a dedicated magnitude control, which can make adaptation to a new numeric scale more direct.

Adapting to a new schema or distribution

For pre-trained models like TabPFN, TabICL, TabSTAR, and FT-Transformer, a new table may have unusual ranges, heavy skew, or odd category sets. A DoRA adapter lets the model re-scale internal features through the magnitude vector while the low-rank part steers patterns. Because the base is frozen and the adapter is small, you can keep one adapter per schema family.

Low rank is attractive for small tables

Many tabular tasks have little labeled data. A low rank limits overfitting. DoRA's tendency to do well at lower ranks suits this regime: a rank of 4 or 8 with a magnitude vector can be enough, with a small file and little risk of memorizing the few rows you have.

In-context learning

In-context learning conditions on labeled rows given at run time. DoRA does not replace it. It complements it: use the context for quick conditioning, and use an adapter when the context alone cannot capture a systematic shift. Because the adapter merges away, the adapted model can be served like any other model and still accept context rows.

Missing values, mixed types, and no column names

Missing-value handling and type embeddings sit in the early layers. Adapting the projection layers there with a magnitude control lets the model change how strongly a missing indicator or a date feature influences the representation. With no column names, the model leans on value statistics, so being able to recalibrate scale per feature direction is useful.

Continual learning

Each new data source gets a separate DoRA adapter over the same frozen base. The base does not change, so earlier adapters remain valid, which avoids catastrophic forgetting. Compare this with weight-regularization approaches such as EWC. For heavier updates, see continued pre-training.

Imputation, anomaly detection, and sector classification

• Imputation: Train a DoRA adapter on masked cells from the target table so that predictions reflect its real missingness. See masked cell modeling and imputation networks.
• Anomaly detection: Adapt the notion of a normal row to one asset or one process using only normal data. See anomaly detection.
• Sector and semantic classification: Adapt the encoder to a new taxonomy or industry vocabulary. See semantic sector heads.

Summary table for tabular use

Honest caveats

• The benefit of DoRA on tabular models has to be measured, not assumed. Reported gains come mostly from language and vision-language work.
• Table models differ in layer structure, so check which layers are linear and eligible.
• Compare against plain LoRA on held-out tables, not just held-out rows.

Practical guidance and best practices

Start from a LoRA recipe

• Use the same rank, alpha, dropout, and target modules that work for LoRA.
• Turn on DoRA and compare.
• Keep learning rate similar at first, then tune.

Rank selection

• Try a lower rank than you would for LoRA, since DoRA may need less.
• Increase the rank only if validation loss plateaus.
• Keep alpha in a sensible ratio to rank. See LoRA and rsLoRA.

Memory and speed

• Expect higher training time than LoRA.
• Use bf16 and gradient checkpointing. See mixed precision and gradient checkpointing.
• Use library versions with optimized DoRA kernels when available.

Quantized bases

• DoRA on a 4-bit base is possible but adds overhead, because the weights must be dequantized for the norm computation.
• Test the combination on a small run before a long one. See QLoRA.

Evaluation

• Run several seeds and report mean and spread.
• Compare against LoRA and, if possible, a small full fine-tuning baseline.
• Test the merged model, not only the adapter model.

Deployment

• Merge for single-task serving.
• If you need to serve many adapters over one base, confirm that your serving engine supports DoRA before choosing it. See vLLM.
• Store the exact base checkpoint identifier with the adapter.

Starting hyperparameters

Diagnostics worth logging

• The mean change in the magnitude vector per layer. If it barely moves, the magnitude control is not doing much for your task.
• The size of the low-rank update relative to the frozen weight. A very large ratio signals a learning rate that is too high.
• Validation loss per layer group if your tooling allows ablations, such as freezing the magnitude vectors to see their effect.

A simple ablation plan

1. Train LoRA with your best settings and record validation metrics over three seeds.
2. Train DoRA with the same settings over the same three seeds.
3. Train DoRA at half the rank to see if the gain survives a smaller adapter.
4. Pick the smallest setup whose mean is within noise of the best one.

This plan favors the cheapest configuration that works, which matters because the extra training cost of DoRA is real.

Tools that support it

• PEFT offers a simple flag.
• Axolotl, LLaMA-Factory, and torchtune expose similar settings in their configs.

Common mistakes

• Expecting a large gain on every task. Improvements can be small or within noise.
• Changing several hyperparameters at once while comparing DoRA with LoRA.
• Using a single seed to declare a winner.
• Forgetting that DoRA costs more training time and then being surprised by slow runs.
• Assuming every serving stack can load unmerged DoRA adapters.
• Combining DoRA with a quantized base without checking memory and speed.
• Initializing the magnitude wrongly in custom code, so the model does not start at the base weights.
• Normalizing along the wrong axis in a custom implementation.
• Using a very high rank and ignoring the scaling issue that rsLoRA addresses.
• Skipping evaluation after merging.
• Not versioning the adapter with its base model.

Frequently asked questions

What is DoRA?

DoRA is weight-decomposed low-rank adaptation. It separates each pre-trained weight into a magnitude and a direction, trains a low-rank update for the direction, and trains the magnitude as a small vector. The goal is to make parameter-efficient fine-tuning behave more like full fine-tuning.

How is DoRA different from LoRA?

LoRA adds a low-rank update directly to the weight, which changes direction and length together. DoRA normalizes the updated weight to get a direction, then multiplies it by a separately learned magnitude. This adds a small number of parameters and some compute but gives the optimizer two simpler controls.

Does DoRA add inference cost?

No, not after merging. Once training ends you compute the final weight and store it as a plain matrix, so the merged model has the same size and speed as the base. If you keep the adapter separate, there is extra computation per layer.

Is DoRA always better than LoRA?

No. Reported results show gains on many tasks, but on easy tasks the difference can be small or hidden by seed noise. You should run a fair comparison with matched settings and several seeds before switching.

Why does separating magnitude and direction help?

Analyses of full fine-tuning suggest that magnitude and direction change in a particular way, while LoRA ties them together more tightly. Giving each its own parameters lets the adapter follow a pattern closer to full fine-tuning. The practical result is often better quality at the same rank.

Can DoRA be used with a quantized model?

Yes, it can be combined with a 4-bit base as in QLoRA. The cost is more overhead, because the method needs the dequantized weights to compute norms. Try a short run first to check memory and speed.

What rank should I use with DoRA?

Start with the rank you would use for LoRA, such as 8 or 16, and also try a lower one. DoRA often holds up at small ranks. Raise the rank only if validation loss stops improving.

Which layers should DoRA target?

The common choice is all linear layers, as with modern LoRA recipes. Targeting only a few attention projections is the smallest setup and can leave quality behind. The right layer names depend on the model, so check them in your library.

Can DoRA be applied to tabular models?

Yes, wherever the model has linear layers that a library can wrap. Its magnitude control may help with numeric scale shifts across tables. Since most published results come from other domains, measure it on held-out tables before relying on it.

How many extra parameters does DoRA add?

One magnitude vector per adapted layer. For large layers this is tiny next to the low-rank matrices, so the total stays close to the LoRA count.

Glossary

• Adapter: A small set of trainable parameters added to a frozen model.
• Column norm: The Euclidean length of a matrix column.
• Direction: A vector of unit length that points along a weight column.
• Fine-tuning: Further training of a pre-trained model on a narrower task.
• Frozen weights: Parameters that do not change during training.
• Low-rank: Expressed through a small number of independent directions.
• Magnitude: The length of a weight column.
• Merge: Folding the adapter into the base weights.
• Normalization: Dividing a vector by its length.
• Rank: The inner size of the low-rank matrices.
• Weight decomposition: Splitting a weight into magnitude and direction.
• Weight normalization: An older method that reparameterizes weights as length times direction.

Summary

DoRA decomposes each frozen weight into magnitude and direction, applies a LoRA-style low-rank update to the direction, and trains the magnitude as a small vector. This decoupling often lets adaptation track full fine-tuning more closely than plain LoRA, especially at low rank, while keeping adapters small and mergeable. The costs are extra training compute, a few more parameters, and uneven support in serving tools.

For tabular foundation models, DoRA offers a way to recalibrate feature scale and steer learned patterns when adapting to new schemas, shifted distributions, imputation tasks, anomaly baselines, and sector classification, with one adapter per task and no forgetting of earlier ones. As always, verify the gain against LoRA on held-out tables. Related reading: LoRA, QLoRA, rsLoRA, GaLore, PEFT, and the benchmarks page.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: