LoRA: fine-tuning with a low-rank update
LoRA fine-tunes a large model by learning a small low-rank update B·A next to each frozen weight matrix. Set the rank of the update the task needs and the rank you give LoRA, then train it.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The three heatmaps are the update the task needs, what LoRA has learned so far, and the remaining error. The chart is the training loss with the best possible rank-r loss from the singular value decomposition; the bars are the singular values of the needed update.
Things to try
- Give LoRA a lower rank than the true rank: the loss stops at the dashed line.
- Match the ranks: the error goes to the noise level.
- Change hidden size and target modules to see real parameter counts.
How it works in real systems
PEFT's LoraConfig(r, lora_alpha, target_modules) wraps the chosen layers; QLoRA keeps the frozen model in 4-bit NF4 so a 70B model can be fine-tuned on one large GPU. Adapters can be merged into the weights for inference.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · LoRA as a layer
Freeze W; learn B (d_out × r) and A (r × d_in); add (α/r)·B·A·x. B starts at zero, so the model starts unchanged.
import torch, torch.nn as nn, math
class LoRALinear(nn.Module):
def __init__(self, base: nn.Linear, r=8, alpha=16):
super().__init__()
self.base = base; self.base.weight.requires_grad_(False)
self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
self.B = nn.Parameter(torch.zeros(base.out_features, r))
self.scale = alpha / r
def forward(self, x):
return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale
def merge(self): # fold the adapter in for inference: no extra latency
self.base.weight.data += self.scale * self.B @ self.A
layer = LoRALinear(nn.Linear(4096, 4096), r=16)
print(sum(p.numel() for p in layer.parameters() if p.requires_grad), "trainable of", 4096 * 4096)
Step 2 · The best rank-r update is the truncated SVD
torch.manual_seed(0)
dW = torch.randn(16, 3) @ torch.randn(3, 16) + 0.05 * torch.randn(16, 16) # an update of rank ~3
U, S, Vh = torch.linalg.svd(dW)
for r in (1, 2, 3, 4):
best = U[:, :r] @ torch.diag(S[:r]) @ Vh[:r]
print(r, "relative error", ((dW - best).norm() / dW.norm()).item())
Step 3 · Fine-tune a real model with PEFT (and QLoRA)
# pip install transformers peft bitsandbytes accelerate
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16) # QLoRA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
model = get_peft_model(model, cfg)
model.print_trainable_parameters() # well under 1% of the parameters
# train with transformers.Trainer or trl.SFTTrainer, then model.save_pretrained("adapter/")