Özer Cem KELAHMETKELAHMET
AI Simulations · Language models

LoRA: fine-tuning with a low-rank update

LoRA fine-tunes a large model by learning a small low-rank update B·A next to each frozen weight matrix. Set the rank of the update the task needs and the rank you give LoRA, then train it.

Interactive simulation. It needs JavaScript; the text below explains what it shows.

What you are looking at

The three heatmaps are the update the task needs, what LoRA has learned so far, and the remaining error. The chart is the training loss with the best possible rank-r loss from the singular value decomposition; the bars are the singular values of the needed update.

Things to try

  • Give LoRA a lower rank than the true rank: the loss stops at the dashed line.
  • Match the ranks: the error goes to the noise level.
  • Change hidden size and target modules to see real parameter counts.

How it works in real systems

PEFT's LoraConfig(r, lora_alpha, target_modules) wraps the chosen layers; QLoRA keeps the frozen model in 4-bit NF4 so a 70B model can be fine-tuned on one large GPU. Adapters can be merged into the weights for inference.

Code lab

The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.

Step 1 · LoRA as a layer

Freeze W; learn B (d_out × r) and A (r × d_in); add (α/r)·B·A·x. B starts at zero, so the model starts unchanged.

import torch, torch.nn as nn, math
class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base; self.base.weight.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale
    def merge(self):                                    # fold the adapter in for inference: no extra latency
        self.base.weight.data += self.scale * self.B @ self.A
layer = LoRALinear(nn.Linear(4096, 4096), r=16)
print(sum(p.numel() for p in layer.parameters() if p.requires_grad), "trainable of", 4096 * 4096)

Step 2 · The best rank-r update is the truncated SVD

torch.manual_seed(0)
dW = torch.randn(16, 3) @ torch.randn(3, 16) + 0.05 * torch.randn(16, 16)   # an update of rank ~3
U, S, Vh = torch.linalg.svd(dW)
for r in (1, 2, 3, 4):
    best = U[:, :r] @ torch.diag(S[:r]) @ Vh[:r]
    print(r, "relative error", ((dW - best).norm() / dW.norm()).item())

Step 3 · Fine-tune a real model with PEFT (and QLoRA)

# pip install transformers peft bitsandbytes accelerate
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)   # QLoRA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
model = get_peft_model(model, cfg)
model.print_trainable_parameters()      # well under 1% of the parameters
# train with transformers.Trainer or trl.SFTTrainer, then model.save_pretrained("adapter/")

More simulations

← Beam search · All AI Simulations · Quantisation →

Last updated Nov 17, 2026
Type to search.
    ↑↓ navigate↵ open