rsLoRA

Overview

rsLoRA, short for rank-stabilized LoRA, is a small but important change to LoRA. It changes the factor that scales the low-rank update. Standard LoRA multiplies the update by alpha divided by the rank r. rsLoRA multiplies it by alpha divided by the square root of r. With that one change, learning stays stable as the rank grows, so a larger rank can actually deliver better results instead of learning more slowly or not at all.

The method adds no parameters, no extra compute, and no change to inference. You can merge the adapter into the base weights exactly as with LoRA. In practice it is a single flag in common libraries such as PEFT. It matters most when you want to use ranks in the dozens or hundreds, for example to give an adapter more capacity on a hard task. At low ranks like 4 or 8, the difference between the two scalings is small, and rsLoRA behaves much like plain LoRA with a rescaled alpha.

What it is

rsLoRA is a scaling rule for LoRA adapters. It does not alter the architecture of the adapter, the initialization, or the training loop. It only changes one constant that multiplies the output of the adapter.

The formula

Standard LoRA computes the layer output as:

y = W x + (alpha / r) * B (A x)

rsLoRA computes:

y = W x + (alpha / sqrt(r)) * B (A x)

Everything else is unchanged. The base weight W is frozen, A is r by d_in, B is d_out by r, B starts at zero, and A starts small and random.

An everyday analogy

Imagine a choir. Each singer is one rank. If you ask a bigger choir to sing and you divide the total volume by the number of singers, each voice becomes quieter as the choir grows, and the whole choir may end up too soft to be heard. If you divide by the square root of the number of singers instead, the choir keeps a steady overall loudness whatever its size. rsLoRA is the second rule. It keeps the sound of the adapter steady as you add more singers.

The problem it solves

With the standard factor, raising the rank often gives little or no gain. People saw that rank 8 and rank 256 produced nearly the same loss curves, which suggested that the extra rank was wasted. The explanation in the rsLoRA work is that the standard factor makes the update shrink too much as r grows, which slows learning and can nearly stop it at high rank. It is not that the high-rank adapter lacks capacity. It is that training cannot use it.

What stays the same

• Parameter count, which depends only on rank and layer size.
• Memory use and speed, apart from tiny constants.
• Merging and inference, which are identical to LoRA.
• Compatibility with QLoRA and DoRA.

Where it came from

The LoRA scaling convention

When LoRA was proposed in 2021, the update was scaled by alpha over r. This was a practical convention. The idea was that alpha could be set once and left alone when the rank changed, which supposedly reduces the need to retune the learning rate for each rank. In reality the convention carries an assumption about how updates grow with rank.

The observation

Practitioners found that going from small to large rank under the standard scaling gave small improvements, or none, and that gradients for high-rank adapters were tiny. This led to a common belief that low ranks are always enough. That belief was partly an effect of the scaling choice.

The rsLoRA paper

The paper "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA" appeared in 2023. It analyzed LoRA adapters in a limit where the learning rate and rank vary and asked which scaling factor keeps the learning stable. It argued that the factor should be proportional to one over the square root of the rank. It called this the rank-stabilized scaling factor and showed experiments in which higher ranks then gave better fine-tuning performance at the cost of more training compute, with no extra inference cost.

Adoption

The method was added to widely used libraries as an option. It is now routinely listed beside DoRA and other LoRA variants in training frameworks such as Unsloth, Axolotl, and LLaMA-Factory.

How it works

Step 1: recall what the adapter outputs

The adapter output is the product of B and A applied to the input, then multiplied by the scaling factor. The size of this output controls how much the adapter changes the layer.

Step 2: think about how the sum grows with rank

B A x is a sum over r terms, one for each rank component. Each term has random-looking contributions at initialization and during early training. When you sum r roughly independent terms of similar size, the sum grows like the square root of r, not like r. This is the same idea as a random walk: after n steps of equal size in random directions, you are about the square root of n steps away from the start, not n.

Step 3: choose the scale to cancel the growth

To keep the adapter output at a steady size, you must divide by the square root of r. Dividing by r, as in standard LoRA, overcorrects. The output then shrinks as r grows, roughly in proportion to one over the square root of r.

Step 4: what overcorrection does to learning

A shrunken output means a shrunken gradient flowing back to A and B during training. Smaller gradients mean slower learning. At high rank the slowdown is large. The adapter has many parameters, yet each step moves the function only a little. This is why high-rank adapters under standard scaling can look no better than low-rank ones.

Step 5: the rank-stabilized fix

With alpha over the square root of r, the output and gradient sizes stay in a stable range as r changes. A larger rank then means more capacity that can be used, and loss usually goes lower with more rank, at the price of more compute.

A walk-through with numbers

Let alpha be 16.

Going from rank 4 to rank 256, the standard factor falls by a factor of 64. The rsLoRA factor falls by a factor of 8. The rsLoRA factor changes much less across ranks, which is the point. Notice also that at the same alpha, rsLoRA factors are larger, so the effective step size is larger. This matters when you migrate a recipe, as discussed below.

The relationship between the two factors

The ratio of the rsLoRA factor to the standard factor is the square root of r. At rank 16 it is 4. At rank 64 it is 8. So switching to rsLoRA at a fixed alpha makes the adapter several times louder than before. To reproduce a known good LoRA run with rsLoRA, you can set alpha to the old alpha divided by the square root of r, which gives the same factor. The benefit then shows up when you raise r.

Looking at the gradient

Training updates A and B using gradients. The gradient with respect to B contains the scaling factor, and the gradient with respect to A contains it as well, multiplied by B. Since B starts at zero, the first updates move B, and A follows once B is nonzero. If the factor is small, the early gradients on B are small, and the whole process starts slowly. With a larger rank under the standard factor, this effect compounds, because every rank component sees the same shrunken signal.

Learning rate and factor are linked

In a simple view, the factor and the learning rate both control how fast the adapter moves the function. Doubling the factor acts a bit like raising the learning rate, though not exactly, because of the two-matrix structure. This is why migrating to rsLoRA at a fixed alpha can feel like a jump in learning rate. It also explains why one flag can look like a big change on some runs and a small one on others.

A thought experiment

Imagine two adapters, one at rank 8 and one at rank 512, both trained with the standard factor and the same alpha. At rank 8 the factor is alpha over 8. At rank 512 it is alpha over 512, which is 64 times smaller. If each rank component contributes equally, the large adapter has 64 times the components but only about 8 times the summed output, so its net output is 8 times smaller than the small one. The large adapter is quieter, not louder. Under the rsLoRA factor the two outputs match in size, which is the aim.

Why this is an analogy to well-known scaling rules

Many neural network parameterization rules scale by one over the square root of the width so that signal size stays steady across layers. Attention scores divide by the square root of the key dimension for the same reason. rsLoRA applies that same logic to the adapter rank.

Key concepts and terms

A worked example

A numeric check of output size

Suppose each of r rank components contributes a value with a typical size of 1, with random signs. The sum of r such terms has a typical size of about the square root of r.

Dividing by r makes the output fade with rank. Dividing by the square root keeps it at 1. This toy table captures the intuition. Real training dynamics are more complex, but the direction of the argument holds.

Enabling it with PEFT

import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained(
    "example-org/base-model", torch_dtype=torch.bfloat16  # placeholder
)

config = LoraConfig(
    r=128,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules="all-linear",
    use_rslora=True,         # alpha / sqrt(r) instead of alpha / r
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
model.print_trainable_parameters()

A rank sweep to test the claim on your own task

ranks = [8, 16, 32, 64, 128, 256]
results = {}

for use_rs in [False, True]:
    for r in ranks:
        cfg = LoraConfig(
            r=r,
            lora_alpha=16,
            target_modules="all-linear",
            use_rslora=use_rs,
            task_type="CAUSAL_LM",
        )
        m = get_peft_model(load_base(), cfg)      # load_base() is your helper
        val_loss = train_and_eval(m, seed=0)      # your training function
        results[(use_rs, r)] = val_loss

for key, loss in sorted(results.items()):
    print(key, round(loss, 4))

You should expect, if the claim holds on your data, that validation loss under standard scaling is flat or slowly improving with rank, while under rsLoRA it improves more clearly until capacity or data limits it. Run a few seeds. Do not expect the same shape on every task.

Migrating an existing recipe

Say your tuned LoRA recipe uses rank 16 and alpha 32. The standard factor is 2. To get the same factor with rsLoRA at rank 16, use alpha equal to 2 times the square root of 16, which is 8. Then, when you increase the rank to 64, the rsLoRA factor becomes 8 over 8, which is 1.0, while the standard factor would have been 0.5 with alpha 32. This shows how rsLoRA keeps the factor steadier as rank rises.

A from-scratch layer

import math
import torch
import torch.nn as nn

class RSLoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=64, alpha=16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / math.sqrt(r)     # the only change from LoRA

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

Strengths and limitations

Strengths

• A one-line change with no extra parameters, memory, or inference cost.
• Makes high ranks useful, so you can trade more training compute for better quality.
• Removes a hidden reason why large ranks seemed not to help.
• Compatible with other LoRA variants and with quantized bases.
• Reduces the need to retune the learning rate when exploring different ranks.
• Backed by a clear mathematical argument.

Limitations

• Benefits appear mainly at medium and high ranks. At rank 4 to 16 the effect is mild.
• Higher rank means more training compute and larger adapter files, even if inference is unchanged.
• The same alpha now gives a much larger step, so old learning rates may be too high.
• Quality is still limited by data and by the base model. More rank does not fix bad data.
• Not every task benefits from high capacity. Easy tasks can be learned at low rank.
• It does not address the other gaps between LoRA and full fine-tuning, such as the coupling of magnitude and direction that DoRA targets.
• Results depend on training length. Very short runs may not show a difference.

When it helps most

• You want to use rank 64 or higher.
• Your task is hard or the dataset is large.
• You noticed that raising rank did not change the loss under standard LoRA.

When it does not matter much

• Small adapters at rank 8 or below.
• Tasks that are already solved at low rank.
• Short, quick experiments where compute is the main limit.

How it compares with alternatives

How rsLoRA relates to other variants

• It can be combined with DoRA. DoRA decides what is learned, rsLoRA decides how it is scaled.
• It can be combined with QLoRA. The scale only changes the adapter output, not the base storage.
• It does not replace tuning the learning rate and alpha.

Decision table

Cost of a higher rank

These ratios apply to the adapter alone. The frozen base dominates total compute for small ranks, so the real overhead of rank 32 is usually small. At rank 512 the adapter is a noticeable part of the cost, and the gain must justify it.

A note on other scaling proposals

Other work has explored different parameterizations and different learning rates for A and B, as well as better initialization. These address related symptoms from different angles. They are not mutually exclusive with rsLoRA, but combining them needs testing. Treat each as a hypothesis to measure on your task.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They take rows and columns with mixed types, deal with missing cells, often work without useful column names, and are pre-trained on many synthetic or real tables. They solve tasks via in-context learning and may be updated over time. See /model/architecture, /model/research, and /model/benchmarks.

Capacity on demand for hard schemas

Some tables are close to what the base model saw in pre-training. Others have strong structure the model has not met, such as deep interactions between many columns or unusual encodings. For the hard ones, you want an adapter with more capacity. rsLoRA is what makes extra rank usable. You can raise the rank for a difficult table family and expect the added rank to help, instead of silently learning at a crawl.

Fewer retuning cycles across rank

Because the factor stays steady, a learning rate that works at one rank is more likely to work at another. When you run many adaptations across customers or schemas, this reduces the search work. You can use a small rank for easy schemas and a larger one for hard ones with the same recipe.

Models such as TabPFN, TabICL, TabSTAR, and FT-Transformer

For pre-trained table models like TabPFN, TabICL, TabSTAR, and FT-Transformer, the linear layers inside the table encoder and the row-interaction blocks are candidates for adapters. Wider layers support larger ranks. The scaling rule matters most where layers are wide and where you choose a large rank.

In-context learning

In-context learning provides labeled rows at run time. A high-rank adapter gives a stronger gradient-based correction on top. The two complement each other. If the context cannot express a shift, such as a new numeric scale or a new interaction between columns, a high-rank adapter can learn it. After merging, the model still accepts context rows as usual.

Mixed types, missing values, and no column names

Different column types often have separate embedding paths. A table with many distinct types and a heavy missing-value pattern needs a richer correction than a table with a few clean numeric columns. Higher rank gives room to adapt several paths at once. Without column names, the model must infer meaning from values, and a richer adapter can encode sector-specific regularities that names would otherwise supply.

Continual learning

Each new data source or period gets a new adapter over the frozen base. rsLoRA lets you give harder periods larger ranks without changing the recipe. The base never changes, so earlier adapters keep working. This is a practical defence against catastrophic forgetting, and a simpler one than weight-regularization methods such as EWC. For heavier updates see continued pre-training.

Imputation, anomaly detection, and sector classification

• Imputation: Train a high-rank adapter with a masked-cell objective on the target table to capture complex missingness patterns. See masked cell modeling and imputation networks.
• Anomaly detection: Tune what normal looks like for one asset class. A larger rank captures more normal modes. See anomaly detection.
• Sector and semantic classification: Adapt the encoder to a rich taxonomy with many labels. See semantic sector heads.

Quick map

Honest caveats

• Table data sets are often small. Very high rank on few rows invites overfitting, so use validation tables and early stopping.
• Many tabular models are small, so the compute cost of high rank is mild, but the benefit may saturate early.
• Published evidence for rsLoRA comes mainly from language models. Measure on your own held-out tables.

Practical guidance and best practices

When to switch it on

• Turn it on when you plan to use rank 32 or above.
• Turn it on when a rank sweep under standard scaling looks flat.
• Leave it off for tiny adapters unless you want consistency across experiments.

Adjusting alpha and learning rate

• Because the same alpha gives a larger factor, start with a smaller alpha or a lower learning rate than your LoRA recipe.
• A safe approach is to match the old factor at your current rank, as in the migration example, then raise the rank.
• Watch the first hundred steps for loss spikes.

Choosing the rank

• Sweep powers of two, such as 16, 32, 64, 128.
• Stop when validation loss stops improving. Higher rank costs more.
• Prefer the smallest rank within noise of the best.

Regularization

• Higher rank can overfit small data. Use dropout, weight decay, early stopping, and held-out validation.
• Use a larger and cleaner dataset before a larger rank if both are possible.

Combining with other methods

• With DoRA, enable both flags and compare to each alone.
• With QLoRA, apply the scaling to adapters over a 4-bit base.
• With gradient checkpointing and mixed precision, the cost of high rank is usually acceptable.

Troubleshooting table

A recommended experiment protocol

1. Fix the data split, seed list, and training length.
2. Choose a base alpha and learning rate from a working LoRA recipe.
3. Convert alpha so the rsLoRA factor matches the old factor at the current rank.
4. Sweep ranks upward and record validation loss and the task metric.
5. Repeat the sweep under standard scaling with the original alpha.
6. Plot both curves. Look for the rank at which each curve stops improving.
7. Pick the smallest rank on the rsLoRA curve that is within noise of the best, and confirm on a held-out set.

What a good result looks like

• The standard-scaling curve is flat or rises slowly with rank.
• The rsLoRA curve falls with rank until it reaches a floor set by data or model limits.
• The floor of the rsLoRA curve is lower than the standard curve at the same high rank.

If both curves are flat and equal, your task does not need high capacity. Use a small rank and save the compute.

Reproducibility notes

• Record the scaling mode in your experiment log, not only the rank and alpha.
• Save the full adapter configuration with the weights.
• Pin library versions, because defaults for scaling and initialization can change between releases.
• Use the same tokenizer, template, and data order across the compared runs.

Evaluation and reporting

• Run several seeds. Report mean and spread.
• Compare against a standard LoRA at matched trainable parameter count and at matched compute.
• Test the merged model.

Deployment

• Merge for single-task serving. The merged model is identical in form to a LoRA merge.
• If you serve unmerged adapters, remember that high-rank adapters are larger and add more compute per request.
• Record whether the adapter used rsLoRA scaling. Loading it with the wrong scaling changes the output. Libraries store this flag in the adapter configuration.

Common mistakes

• Switching on rsLoRA without changing alpha or the learning rate, and then seeing instability because the factor became much larger.
• Judging rsLoRA at rank 8 and concluding it has no effect.
• Comparing rsLoRA at rank 128 to LoRA at rank 128 using different learning rates, and attributing all differences to the scaling.
• Using very high ranks on tiny datasets and overfitting.
• Loading an rsLoRA adapter as if it were standard LoRA, or the reverse, which changes the scale of the update.
• Expecting rsLoRA to fix poor data or a poor base model.
• Forgetting that more rank means more training compute.
• Believing that rank stabilization removes the need to tune anything.
• Ignoring that the claim is theoretical in a limit, and that real runs need verification.
• Not saving the scaling choice with the adapter metadata.

Frequently asked questions

What is rsLoRA?

rsLoRA is rank-stabilized LoRA. It is the same adapter method as LoRA, but the scaling factor on the update is alpha over the square root of the rank instead of alpha over the rank. This keeps training stable as the rank grows and allows higher ranks to improve results.

Why does standard LoRA scaling hurt at high rank?

Dividing by r makes the adapter output shrink as the rank increases, because the sum of r roughly independent terms grows only like the square root of r. The smaller output means smaller gradients, so learning slows. In practice this made large ranks look useless.

Does rsLoRA change inference cost?

No. After training, the scaled low-rank product is merged into the base weight just as in LoRA. The merged model has the same size and speed as the base. Only the training compute rises with a higher rank.

Do I need to change my learning rate?

Often yes. At the same alpha, the rsLoRA factor is larger than the standard factor by the square root of the rank, so the effective step is larger. Start with a lower alpha or learning rate, or set alpha so that the factor matches your old recipe at your current rank.

Is rsLoRA useful at low ranks?

Only slightly. At ranks like 4 or 8 the two factors differ by a small constant, which a different alpha can absorb. The method shows its value when you go to medium and high ranks.

Can I combine rsLoRA with DoRA or QLoRA?

Yes. The scaling factor is independent of how the base is stored and of how the weight is decomposed. Combine them and compare against each alone, because gains do not always add up.

How do I turn it on?

In common libraries it is a configuration flag on the LoRA settings. You set the rank and alpha as usual and enable the rank-stabilized option. Check the library documentation for the exact name in your version.

Will a higher rank always give better results?

No. Rank helps until the data, the base model, or the optimization becomes the limit. On small data, a high rank can overfit. Use a validation set and stop raising the rank when it no longer helps.

Is the square root rule proven?

The rsLoRA paper gives a mathematical argument that a factor proportional to one over the square root of the rank is the one that stabilizes learning in a particular analysis, and it reports experiments that support it. As with any theory, real tasks should be checked empirically.

Does it apply to models other than language models?

Yes. The scaling rule concerns the adapter structure, not the data type, so it applies to any model where LoRA adapters are used, including vision, speech, and tabular models.

Glossary

• Adapter: A small trainable module added to a frozen model.
• Alpha: The numerator in the LoRA scaling factor.
• Capacity: The amount of change an adapter can represent.
• Effective learning rate: The true step size in function space, shaped by the scaling factor and the optimizer.
• Fine-tuning: Further training of a pre-trained model on a narrower task.
• Gradient: The signal that tells parameters how to change.
• Low-rank: Built from a small number of independent directions.
• Merge: Folding the adapter into the base weights.
• Rank: The inner size of the adapter matrices.
• Rank stabilization: Choosing the scaling so training is stable for any rank.
• Scaling factor: The constant that multiplies the adapter output.
• Square root scaling: Dividing by the square root of a size to keep sums at a steady level.

Summary

rsLoRA changes one constant in LoRA: the update is scaled by alpha over the square root of the rank rather than alpha over the rank. This keeps the adapter output and gradient sizes steady as rank grows, so larger ranks learn at a healthy pace and can deliver better quality. The method adds no parameters and no inference cost, and it combines with DoRA and QLoRA. Its main costs are more training compute at high rank and the need to adjust alpha or the learning rate when migrating a recipe.

For tabular foundation models, rsLoRA is a practical way to give difficult schemas, complex missingness, rich normal behavior, or large label sets the extra adapter capacity they need, while keeping one simple recipe across ranks and keeping the frozen base intact for continual learning. As always, verify on held-out tables. Related reading: LoRA, GaLore, PEFT, and the research page.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: