Fine-tuning

Overview

Fine-tuning means taking a model that has already been trained on a large general dataset and training it a little more on a smaller dataset that represents what you actually need. The pretrained model brings broad knowledge and skills. Fine-tuning adjusts its behaviour: following your output format, using your domain's vocabulary, classifying your categories, or answering in the tone your users expect.

The word covers a wide family of techniques. Some update every weight in the model. Some freeze the original weights and train a small add-on. Some teach the model to follow instructions from example answers, and others teach it which of two answers people prefer. This page is the map. Each method has its own detailed page, linked below.

An everyday picture: a qualified chef joining a new restaurant does not go back to cookery school. They already know how to cook. They spend a few weeks learning this kitchen's menu, plating and house style. Fine-tuning is that induction period. Full fine-tuning is like retraining the chef's every habit; parameter-efficient fine-tuning is like handing them a small notebook of house rules that changes how they apply what they already know.

What it is

The first six rows describe how much of the model you change. The last three describe what signal you train on. They combine: a common recipe is LoRA (how) applied to SFT data, then DPO (what) with LoRA again. The collection of cheap methods is known as parameter-efficient fine-tuning, or PEFT.

Where it came from

Fine-tuning grew out of transfer learning in computer vision. Through the 2010s, networks pretrained on ImageNet were routinely fine-tuned for new image tasks; Yosinski and colleagues' 2014 paper "How transferable are features in deep neural networks?" studied why this works. In language, Jeremy Howard and Sebastian Ruder's ULMFiT (2018) showed that a pretrained language model fine-tuned in stages could set new results on text classification. BERT (Devlin and colleagues, 2018) made "pretrain, then fine-tune" the default for language understanding.

As models grew, updating every weight became expensive, which led to parameter-efficient methods. Houlsby and colleagues introduced adapters in 2019. Li and Liang proposed prefix tuning in 2021, and Lester, Al-Rfou and Constant proposed prompt tuning the same year. Edward Hu and colleagues introduced LoRA in 2021. Liu and colleagues introduced IA3 in 2022, in a paper titled "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning". Tim Dettmers and colleagues' QLoRA (2023) showed a 65-billion-parameter model could be fine-tuned on a single 48 GB GPU.

On the behaviour side, Wei and colleagues' FLAN paper (2021) showed that fine-tuning on many tasks phrased as instructions improves zero-shot performance. Ouyang and colleagues' InstructGPT paper (2022) combined supervised fine-tuning with reinforcement learning from human feedback. Rafailov and colleagues' Direct Preference Optimization (2023) removed the separate reward model and reinforcement learning loop, and many variants followed, including ORPO, KTO and SimPO in 2024. Gururangan and colleagues' "Don't Stop Pretraining" (2020) showed the value of continued pre-training on domain text.

How it works

The common core

All fine-tuning is ordinary gradient-based training that starts from pretrained weights instead of random ones. You choose a loss, feed your data in batches, and update the trainable parameters with an optimiser such as AdamW. What differs between methods is which parameters are trainable and which loss is used.

Full fine-tuning:   θ ← θ − η · ∇_θ L(θ; D)
LoRA update:        W' = W₀ + (α / r) · B · A,     B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), r ≪ min(d, k)
SFT loss:           L_SFT = − Σ_t log p_θ(y_t | x, y_<t)
DPO loss:           L_DPO = − log σ( β · [ log(p_θ(y_w|x) / p_ref(y_w|x)) − log(p_θ(y_l|x) / p_ref(y_l|x)) ] )

In the first line θ is the set of trainable parameters, η the learning rate, L the loss and D your dataset. In the LoRA line W₀ is a frozen pretrained weight matrix of size d × k, A and B are the small trainable matrices, r is the rank (often 8 to 64), and α is a scaling constant. In the SFT loss, x is the prompt, y_t the t-th token of the target answer and y_<t the tokens before it; the loss is computed only on the answer tokens. In the DPO loss, y_w is the preferred answer, y_l the rejected one, p_ref the frozen starting model, σ the sigmoid function and β a strength setting that controls how far the model may move from the reference.

Full fine-tuning

Every weight is updated. This gives the most capacity to change but needs memory for the weights, their gradients and the optimiser state, roughly three to four times the model size or more in mixed precision. It also produces a full new copy of the model per task, and it is the most prone to forgetting general skills.

Parameter-efficient fine-tuning

PEFT methods freeze the pretrained model and train a small number of new parameters, often well under one per cent of the total. LoRA learns a low-rank correction to selected weight matrices, typically the attention and feed-forward projections. After training the correction can be merged into the weights, so inference costs nothing extra. QLoRA stores the frozen base in a 4-bit format (NF4), uses double quantisation of the scaling constants and paged optimisers to handle memory spikes, and trains LoRA matrices in higher precision. Adapters insert small bottleneck layers; prefix and prompt tuning prepend trainable vectors to the input or to each layer's keys and values; IA3 learns three vectors that rescale keys, values and feed-forward activations. Variants such as DoRA and rsLoRA refine LoRA's parameterisation and scaling.

Instruction tuning and SFT

Supervised fine-tuning trains on pairs of prompts and ideal responses, usually formatted with the model's chat template. The model learns to follow instructions, keep to formats such as JSON, use tools, and adopt a voice. Quality matters more than quantity: the LIMA paper (Zhou and colleagues, 2023) reported strong results from about 1,000 carefully chosen examples, though that depends heavily on the base model and task.

Preference tuning

SFT shows the model good answers but not what makes one answer better than another. Preference tuning uses comparisons. In RLHF, a reward model is trained on human rankings, then the language model is optimised against it with reinforcement learning, typically PPO, with a penalty for drifting too far from the starting model. DPO reaches a similar goal with a single classification-style loss on preferred and rejected pairs. KTO works with single answers labelled good or bad, and ORPO folds preference learning into the SFT step.

Continued pre-training

When the gap is knowledge, not behaviour, for example a new language, legal corpus or codebase, you continue the original next-token objective on large amounts of unlabelled domain text. This is closer to pre-training than to task tuning, needs much more data and compute, and is usually followed by SFT to restore instruction following.

Data preparation

Good data is most of the work. Remove duplicates and near-duplicates, strip personal data you are not allowed to use, balance categories, and make sure the format at training time matches the format at inference time exactly, including the chat template and special tokens. Keep a held-out set that reflects real use, and never let it leak into training.

Hyperparameters

Typical starting points: for LoRA, a learning rate around 1e-4 to 2e-4, rank 8 to 32, alpha equal to or double the rank, and one to three epochs; for full fine-tuning of large models, much smaller learning rates, often around 1e-5 or lower. Use warm-up and a decaying learning rate schedule. For DPO, β around 0.1 is a common default. Treat all of these as starting points to tune, not rules.

Catastrophic forgetting

Training hard on a narrow dataset can erase general skills, a problem first described for neural networks by McCloskey and Cohen in 1989. Remedies include lower learning rates, fewer epochs, mixing in general data, regularising towards the original weights as in elastic weight consolidation (EWC), and using PEFT. Biderman and colleagues' 2024 study "LoRA Learns Less and Forgets Less" found that LoRA typically adapts less than full fine-tuning on hard new domains but preserves more of the base model's abilities.

Evaluation

Evaluate on three things: the target task (held-out examples, task metrics, or LLM-as-judge with a clear rubric), regressions on general abilities you still need, and safety behaviour. Compare against the base model with good prompting, since that is the honest baseline.

Key concepts and terms

A worked example

An illustrative support team wants a 7-billion-parameter open model to turn customer emails into a fixed JSON ticket with fields for product, urgency, category and a one-line summary.

1. Baseline: with a careful prompt and three examples, the base model produces valid JSON 88 per cent of the time and the right category 74 per cent of the time on 500 held-out emails.
2. Data: the team collects 6,000 past emails with corrected tickets, removes duplicates and personal data, and formats each pair in the model's chat template. They hold out 500 for testing.
3. Method: QLoRA, because they have one 24 GB GPU. Rank 16, alpha 32, LoRA on all attention and feed-forward projections, learning rate 2e-4, two epochs, loss on the response only.
4. Result: valid JSON rises to 99.6 per cent and category accuracy to 89 per cent. A short general-ability check shows no meaningful drop.
5. Next: they collect pairs where staff preferred one summary over another and run a light DPO pass with β = 0.1 to improve summary quality.

The figures are illustrative. A minimal LoRA setup with the Hugging Face PEFT library looks like this:

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("your-base-model")
config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
                    task_type="CAUSAL_LM")
model = get_peft_model(model, config)
model.print_trainable_parameters()   # a small fraction of the total

Module names such as q_proj vary between model families, so check them for your base model.

Strengths and limitations

Strengths:

• Changes behaviour reliably: formats, tone, task skill and tool use.
• Shorter prompts at inference, which can lower cost and latency.
• PEFT makes adaptation affordable on a single GPU and lets you swap task adapters.
• Can turn a smaller model into a strong specialist that rivals a larger general one on one task.
• Preference tuning aligns outputs with judgements that are hard to write as rules.

Limitations:

• Needs good labelled data, which is often the expensive part.
• Weak at adding fresh or frequently changing facts; knowledge goes stale.
• Can cause forgetting or weaken safety behaviour if done carelessly.
• Adds a training and evaluation pipeline to maintain.
• Results are sensitive to data format, learning rate and number of epochs.

How it compares with alternatives

A practical rule: use retrieval for what the model should know, fine-tuning for how it should behave, and prompting first for both, because it is cheapest to try. Many systems use all three.

Practical guidance and best practices

1. Exhaust good prompting and a few examples first, and record that score as the baseline.
2. Decide whether the gap is knowledge (use retrieval or continued pre-training) or behaviour (use fine-tuning).
3. Start with LoRA or QLoRA; move to full fine-tuning only if PEFT clearly plateaus.
4. Spend most effort on data quality: clean, deduplicated, consistent and in the exact inference format.
5. Mask the loss to response tokens for SFT and use the model's own chat template.
6. Keep learning rates conservative and epochs few; watch validation loss for overfitting.
7. Evaluate the target task, general abilities and safety separately, every run.
8. Version datasets, configs and adapters together so any result can be reproduced.
9. Add preference tuning only after SFT works, and only with good comparison data.

Common mistakes

• Fine-tuning to add facts that change weekly, instead of using retrieval.
• Training with one prompt format and serving with another.
• Letting test examples leak into the training set through duplicates.
• Training for many epochs on a small set and mistaking memorisation for skill.
• Judging success only on the target task and missing regressions elsewhere.
• Skipping the base-model baseline, so nobody knows whether fine-tuning helped.

Frequently asked questions

How much data do I need to fine-tune?

It depends on the task and base model. Format and style changes can work with a few hundred high-quality examples. Teaching a new skill or domain often needs thousands to tens of thousands. Continued pre-training needs far more, typically measured in millions or billions of tokens. Quality and consistency matter more than raw count.

Is LoRA as good as full fine-tuning?

Often close for behaviour and format tasks, and it forgets less. On hard new domains such as code or maths with large datasets, full fine-tuning can learn more. Because LoRA is much cheaper, it is the usual first choice, with full fine-tuning kept for cases where LoRA clearly falls short.

Should I fine-tune or use RAG?

Use retrieval when the model needs facts it was not trained on, facts that change, or answers that must cite sources. Use fine-tuning when the model knows enough but behaves wrongly, such as ignoring formats or using the wrong tone. The two combine well: a fine-tuned model can be better at using retrieved context.

What is the difference between SFT and instruction tuning?

Instruction tuning is SFT on data made of instructions and responses, often across many tasks. SFT is the broader term for training on any input and target pairs with a standard supervised loss. In LLM practice the two terms are frequently used interchangeably.

Why does DPO need a reference model?

The reference model anchors training. DPO's loss rewards increasing the probability of preferred answers relative to the reference, not in absolute terms. Without that anchor the model could drift far from its starting point and lose fluency or general skills. The β setting controls how strong the anchor is.

Can fine-tuning remove safety behaviour?

Yes. Studies have shown that even small amounts of fine-tuning can weaken a model's refusals and safety habits, sometimes unintentionally. Include safety evaluations in every run, and mix in data that preserves the behaviours you need.

Does fine-tuning apply to tabular models too?

Yes. Many tabular foundation models predict in context without training, but some can also be fine-tuned on a family of related tables to specialise them. The same ideas apply: start from pretrained weights, use a held-out set, and watch for forgetting.

Glossary

• Fine-tuning: further training of a pretrained model on task or domain data.
• PEFT: methods that train only a small number of added parameters.
• LoRA: a low-rank update added to frozen weight matrices.
• QLoRA: LoRA training on a 4-bit quantised frozen base model.
• Adapter: a small trainable module inserted inside each layer.
• SFT: supervised fine-tuning on input and ideal output pairs.
• RLHF: optimising a model against a reward model learned from human preferences.
• DPO: a direct loss on preferred and rejected answer pairs.
• Continued pre-training: more next-token training on unlabelled domain text.
• Catastrophic forgetting: loss of prior skills after new training.

Summary

Fine-tuning adapts a pretrained model by training it further on data that reflects your task. The map has two axes. One is how much of the model you change: everything in full fine-tuning, or a small add-on in parameter-efficient methods such as LoRA, QLoRA, adapters, prefix and prompt tuning and IA3. The other is what signal you train on: raw domain text in continued pre-training, ideal answers in SFT, or comparisons in RLHF, DPO and related methods. Success depends mostly on clean data in the exact inference format, conservative hyperparameters, and evaluation that checks the target task, general skills and safety against a well-prompted baseline. Use retrieval for changing facts, fine-tuning for behaviour, and prompting first, because it is the cheapest experiment.

Related

It belongs to Fine-tuning in my stack, under Generative AI and LLMs. Others in the same group: