Full fine-tuning

Overview

Full fine-tuning means continuing to train a pre-trained model while every one of its parameters is allowed to change. Nothing is frozen and nothing is added on the side. The model is trained on new data with the same kind of loss and optimizer used in pre-training, usually at a much lower learning rate. The result is a complete new copy of the weights that is specialized for the new task or domain.

It is the most flexible way to adapt a model, and it often gives the best quality when you have enough data and compute. The price is high memory use, large storage for each variant, and a higher risk of forgetting what the model knew before. Cheaper options such as LoRA, QLoRA and DoRA train only a small part of the model. This article explains when the full approach is worth it, how it works, how to run it, and how it relates to models that learn from raw tables.

What it is

In a neural network, the parameters are the numbers, called weights and biases, that are learned during training. A large model can have billions of them. In full fine-tuning, the training loop computes a gradient for each parameter and the optimizer updates each parameter. Compare this with parameter-efficient methods, where the original weights stay fixed and only a small add-on is trained.

An analogy helps. Imagine a house. Full fine-tuning is a full renovation where any wall, pipe or wire can be changed. A parameter-efficient method is adding a new room or changing the furniture while the structure stays the same. The renovation can produce a better result for a new purpose, but it costs more, takes more space for materials, and can damage something that worked before.

Where it sits in the training pipeline

Full fine-tuning is a way to run several stages, not a stage by itself.

So when someone says "we fully fine-tuned the model", the question is what data and what objective they used. The phrase describes which weights are trainable, not what the model learned.

Variants within the full approach

• Single-task full fine-tuning. One model per task. Simple, but you store many copies.
• Multi-task full fine-tuning. One model trained on a mix of tasks. Fewer copies, more care with data balance.
• Layer-wise learning rates. Lower rates for early layers and higher for later layers, since early layers hold more general features.
• Gradual unfreezing. Start with only the top layers trainable and unfreeze lower ones step by step. This sits between the full and partial approaches.

Where it came from

Fine-tuning grew out of transfer learning. In computer vision in the 2010s, researchers found that networks trained on large image sets learned features that worked well on other tasks. Teams copied such a network, replaced the last layer and kept training everything on a small dataset. That is full fine-tuning in its early form.

In language, the pattern took hold with pre-trained transformers. Models such as BERT and GPT were released as general encoders or decoders and were fine-tuned end to end on tasks such as classification and question answering. For several years, this was the standard way to use a pre-trained language model, and each task produced a separate full copy.

As models grew from millions to billions of parameters, the cost of storing and training full copies became a problem. Adapter methods, prefix methods and then LoRA appeared to cut that cost, see adapters, prefix tuning, prompt tuning and BitFit. Memory-saving training systems such as DeepSpeed ZeRO, FSDP, Megatron-LM and Colossal-AI made full updates of very large models possible by spreading states across many devices. Recent methods such as GaLore try to keep the full-parameter learning while shrinking optimizer memory. Studies comparing the approaches generally found that the low-rank methods match full updates on many tasks, but that full updates can learn more and forget more when the new data is large and far from the original training data.

How it works

Step 1: Start from pre-trained weights

You load a model that has already been trained. The weights hold what it learned about language, images or tables. This is the starting point, like a student with a degree who now learns a specialty.

Step 2: Prepare data and a loss

You choose data that matches the target job and a loss that fits it, such as next-token cross-entropy for text or cross-entropy on class labels for classification. For instruction data, see the masking and template notes in SFT.

Step 3: Run the forward pass

The model processes a batch and produces predictions. During this pass it stores intermediate results called activations, which are needed to compute gradients later.

Step 4: Compute gradients for all parameters

Backpropagation measures how much each parameter contributed to the error. In full fine-tuning, every parameter gets a gradient. This is the main difference from adapter methods, where most parameters are frozen and get no gradient.

Step 5: Update with the optimizer

An optimizer such as AdamW uses the gradient and its own running statistics to change each weight. AdamW keeps two extra numbers per parameter, a running mean and a running variance of the gradient. These are the optimizer states, and they are the big memory cost.

Step 6: Repeat with a careful schedule

The loop repeats over many batches. A learning-rate schedule starts with a short warmup, then decays. The rate is small, since large steps can erase useful knowledge.

Step 7: Validate, select and save

You check a validation set, compare checkpoints, and keep the best one. The output is a full set of weights, the same size as the original model.

The memory budget in plain numbers

For a model with N parameters trained in mixed precision with AdamW, a common estimate of memory per parameter is:

So a model with 7 billion parameters needs about 112 billion bytes, which is about 112 GB, for the model states alone, before counting activations. This is why a single consumer card cannot do full fine-tuning of a mid-size model without tricks. Sharding the states across devices with ZeRO or FSDP, using 8-bit optimizers, offloading to CPU memory, or using gradient checkpointing reduces the need. Exact numbers vary by setup, so treat the table as an estimate.

Why learning rate matters so much

Think of a sculptor who has finished a statue and now wants a small change. A heavy hammer will break the work. A fine chisel changes the surface. A high learning rate is the hammer. With all weights trainable, a high rate can quickly destroy the pre-trained structure, which is called catastrophic forgetting, see EWC and LwF. Typical fine-tuning rates are one to two orders of magnitude lower than pre-training rates.

Key concepts and terms

A worked example

This example fine-tunes all weights of a small model with the Hugging Face Trainer. Names are placeholders. It uses gradient checkpointing and a low learning rate.

import torch
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          TrainingArguments, Trainer,
                          DataCollatorForLanguageModeling)
from datasets import load_dataset

name = "example-org/small-base-1b"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.bfloat16)

# Every parameter is trainable (this is the default)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"trainable: {trainable} of {total}")

ds = load_dataset("json", data_files="data/domain.jsonl", split="train")
ds = ds.map(lambda r: tok(r["text"], truncation=True, max_length=1024),
            remove_columns=ds.column_names)

args = TrainingArguments(
    output_dir="out/full-ft",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    learning_rate=1e-5,
    num_train_epochs=1,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=20,
    save_steps=500,
)

Trainer(model=model, args=args, train_dataset=ds,
        data_collator=DataCollatorForLanguageModeling(tok, mlm=False)).train()

Scaling it across GPUs with sharding

For larger models you add a sharding config. A minimal DeepSpeed ZeRO stage 3 config looks like this.

# ds_config_zero3.json shown here as YAML for readability
zero_optimization:
  stage: 3
  overlap_comm: true
  contiguous_gradients: true
  offload_optimizer:
    device: cpu
bf16:
  enabled: true
gradient_accumulation_steps: 16
train_micro_batch_size_per_gpu: 2

A numeric look at forgetting

Suppose a model scores 80 percent on a general test before tuning. You fine-tune on a narrow set. After one epoch at a low rate, the narrow task rises from 55 to 85 percent and the general score drops to 78. After three epochs at a high rate, the narrow task reaches 90 but the general score falls to 60. The numbers are illustrative, but the pattern is typical: more aggressive full updates buy more specialization at the price of more forgetting. The fix is to track both numbers, stop early, lower the rate, and mix in general data.

A simple comparison run

To decide between full and low-rank updates, run both on the same data and evaluation. Keep everything equal: data, steps, seed. Record quality, peak memory, time and checkpoint size.

Designing the training data mix

With full updates, the data mix decides what the model becomes. Every weight can move, so every pattern in the data can leave a mark.

Mixing domain data with general data

A common recipe blends the new domain data with a slice of general data. The general slice acts as an anchor. It keeps the model close to its earlier behavior on broad tasks. Teams often start with a ratio of roughly seven parts domain to three parts general, then adjust after looking at general benchmark scores.

Ordering and curriculum

The order of examples can matter. Placing easier or cleaner data first and harder data later is a form of curriculum learning. Random shuffling is a safe default. If you use a curriculum, plan the schedule with care, see curriculum scheduling.

Data quality filters

Large domain corpora contain noise. Before training, remove duplicates, broken records and boilerplate, see deduplication. Full updates are more willing than adapters to absorb whatever is in the data, including mistakes, so a clean set pays back more here.

Synthetic data

When labeled data is scarce, generated data can help, see synthetic data generation. Keep a real validation set that no synthetic example was derived from. Synthetic data has a distribution of its own, and a model can learn quirks that do not exist in real use.

Evaluating a fully tuned model

An evaluation plan for full updates has three parts.

1. Target metric. The score on the task you tuned for, on a held-out set.
2. Retention metric. A set of general tasks that the base model handled well, to detect forgetting.
3. Behavior checks. Format, safety and refusal behavior, since all of these can drift.

Report all three for the base model and for each checkpoint. A table like the one below makes trade-offs visible.

Pick the checkpoint with the best balance, which is often not the last one. Use evals and LLM-as-judge as helpers, and review samples by hand.

Merging and averaging weights

Because a fully tuned model has the same shape as the base, you can combine weights. Simple averaging of checkpoints from one run often improves stability. Averaging two models tuned from the same base on different tasks can give a model that does both reasonably well. The idea works because models tuned from one starting point tend to stay in a connected region of weight space. Merging is not magic. Test the merged model, since conflicts between tasks can still hurt.

A related idea is to compute a task vector, the difference between tuned and base weights, and add a scaled version of it to the base. This gives a way to strengthen or weaken a skill, and to combine several skills, with no extra training. It is a cheap trick, but it needs evaluation after each merge.

Troubleshooting guide

Cost thinking

Cost comes from device hours, storage and the time of the people who prepare data and evaluate. Full updates raise the first two. A fair comparison with a low-rank method should count the extra devices needed, the longer runs, and the storage and serving cost of each full copy. If you plan many variants, for example one per customer or per sector, the storage and serving cost can dominate, and the adapter route usually wins. If you plan one important model that will serve many requests, spending more on training to get a better single model can be the right trade.

Strengths and limitations

Strengths

• Highest capacity to adapt. Every layer can change, so the model can learn new patterns that a small add-on cannot express.
• No extra inference cost. The result is a normal model, with no adapter to merge or load.
• Best for large, distant data. When you have millions of examples from a domain unlike the original data, full updates tend to use that data best.
• Simple mental model. There is nothing extra to configure, such as rank or target modules.
• Works with any architecture, including table-native networks.

Limitations

• Memory heavy. Optimizer states and gradients for all parameters must be stored.
• Storage heavy. Each variant is a full copy of the weights.
• Slower to iterate. Each experiment costs more time and money.
• More forgetting. All weights can drift, so earlier skills may degrade.
• Sensitive to settings. Learning rate, warmup and data order all matter more.
• Overfits small datasets. With few examples, a model with billions of free parameters will memorize them.
• Harder to serve many variants. Hosting dozens of full copies is costly compared with swapping adapters.

How it compares with alternatives

A practical order is to try prompting, then a parameter-efficient method, and move to full updates only if a measured gap remains and the budget allows. Tools such as PEFT, TRL, Axolotl, LLaMA-Factory, torchtune and Unsloth support both full and low-rank training.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed numeric, categorical, text and date fields, missing values, and often no useful column names. Table-native models such as TabPFN, TabICL, TabSTAR and XTab are pre-trained on many tables, frequently synthetic ones drawn from random causal programs, and are then applied through in-context learning: you give labeled rows as context and the model predicts for new rows without a gradient step. Full fine-tuning enters this picture in several concrete ways. See /model/research and /model/architecture for how such models are built.

Pre-training itself is full training

The pre-training of a table-native model updates all its weights on a large stream of tables. In that sense a tabular foundation model is the product of full-parameter training. Continuing that run on a new mix of tables, for example more real-world tables or harder synthetic priors, is a form of full continued training, see continued pre-training.

Specializing a pre-trained table model

When a user has a large set of tables from one sector, such as telecom or manufacturing, and in-context use is not enough, full fine-tuning can adapt the whole network. It lets row attention, column attention and the encoders for numeric and categorical cells shift toward the sector's statistics, see axial attention and FT-Transformer. The gain is usually largest when the sector's data has unusual distributions, heavy skew or many rare categories.

Handling mixed types, missing values and unnamed columns

Table models have input modules for numbers, categories and sometimes text. With full updates these modules can be retuned. For example, a numeric encoder can learn better scaling for heavy-tailed fields, and the missing-value embedding can learn how gaps behave in the target sector. To keep the model robust to unnamed columns, include training tables where headers are removed or shuffled, so the model keeps relying on the values and their statistics.

Imputation, anomaly detection and sector classification

Several downstream tasks benefit from supervised full updates.

Continual learning

A table model that is fully fine-tuned for a second sector may lose accuracy on the first. Ways to reduce this include mixing replay tables from earlier sectors, using lower learning rates, regularizing important weights with EWC or synaptic intelligence, and keeping small per-sector adapters instead of new full copies. For multi-sector products, a common design is one frozen base plus per-sector adapters, with full updates reserved for periodic base refreshes.

When not to use it for tables

Table models with in-context learning already adapt to a new table in one forward pass, with no training and no risk of forgetting. For most single-dataset jobs, that is the right default. Reserve full fine-tuning for cases where a large labeled corpus exists, the schema is stable, and a held-out comparison shows a clear gain. See /model/benchmarks for how such comparisons are set up.

Practical guidance and best practices

Decide first, then train

• Write down the target metric and the baseline: prompting, few-shot, in-context, and LoRA.
• Estimate the memory and time budget before starting. Use the table of bytes per parameter above.
• Choose full updates only if the gap to cheaper methods justifies it.

Settings that usually work

• Use a low learning rate and a warmup. Start near 1e-5 for mid-size models and lower for larger ones, then tune.
• Keep epochs few. One to three passes are common for instruction data, and a single pass for large corpora.
• Use bf16 where the hardware supports it, with 32-bit master weights in the optimizer.
• Use gradient accumulation for a large effective batch.
• Turn on gradient checkpointing if activations do not fit.
• Use sharding such as ZeRO or FSDP when the model states exceed one device.
• Fix seeds and log configs, so results can be reproduced.

Protect what the model knew

• Mix in general or earlier-domain data, often ten to thirty percent.
• Evaluate on a general benchmark during training, see lm-evaluation-harness.
• Keep the learning rate lower for lower layers if forgetting appears.
• Consider freezing embeddings or the first layers if the data is small.
• Average or merge checkpoints if training was noisy.

Operations

• Track runs and artifacts, see MLflow and Weights & Biases.
• Save checkpoints often. Long full runs fail for ordinary reasons such as hardware faults, and you want to resume.
• Verify the saved model by loading it fresh and re-running the evaluation.
• Quantize the final model for serving if latency matters, see AWQ, GPTQ and GGUF.
• Serve with an efficient runtime such as vLLM or TensorRT.

Hardware notes

Full runs benefit from fast links between devices because sharding moves states often, see NCCL and CUDA. Accelerators such as NVIDIA H100, H200 and B200 have more memory per device, which reduces how much sharding you need. Cloud accelerators, see Cloud TPU, are another route.

A short checklist before pressing start

• The baseline numbers for prompting and a low-rank run are recorded.
• The validation and retention sets are fixed and not used for tuning.
• The data is deduplicated and split by group.
• The template and tokenizer match inference.
• The memory estimate fits the hardware, with a margin.
• Checkpoints and logs go to storage that survives a restart.
• A stop rule exists, such as a limit on epochs or a patience value.

Partial freezing as a middle path

Full updates do not have to be all or nothing. You can freeze parts of the model and train the rest.

Each choice lowers memory and forgetting risk, and also lowers the ceiling a little. Measure before you decide.

Optimizer choices

AdamW is the default. Variants reduce its memory: 8-bit optimizer states, see bitsandbytes, factored second moments, or projecting gradients to a low-rank space as in GaLore. Simpler optimizers such as SGD with momentum use less memory but are harder to tune for large transformers. If memory is the only problem, try an 8-bit optimizer before changing the method.

Reproducibility

Full runs are long and costly, so being able to repeat them matters. Record the code version, data version, config, seed, library versions and hardware. Note that distributed runs may not be bit-exact even with fixed seeds. Compare results across seeds when a decision depends on a small difference.

Governance and audit

A fully tuned model is a new artifact. Store its lineage: base model, data sources, filters, and evaluation results. This helps with audits, privacy requests and rollbacks, see data lineage and data contract validation. If the training data contained personal information, document how it was handled, see GDPR and KVKK.

Rollout and rollback

Ship a fully tuned model like any other release. Run it in shadow mode first, compare its answers with the current model on live traffic, and release to a small share of users. Keep the previous weights available for a fast rollback. Because full copies are large, plan storage and loading time ahead, since swapping a multi-gigabyte model in production is not instant.

When the base model is updated

Teams that fully tune a base model face a maintenance problem. When a new base version appears, the tuned weights do not transfer, and the whole run must be repeated on the new base. Keep the data, filters and configs in a form that lets you rerun the recipe with little manual work. This is another reason adapters and data-centric recipes are attractive: the expensive asset is the dataset, not the weights.

Common mistakes

• Learning rate too high. The most common cause of ruined models. Pre-training rates are wrong for fine-tuning.
• No general evaluation. Without a general check, forgetting goes unnoticed until users complain.
• Underestimating memory. Counting only the weights and forgetting gradients and optimizer states leads to out-of-memory errors.
• Training on a tiny set. A few hundred examples are rarely enough to justify updating billions of parameters. Use a low-rank method or in-context learning.
• Skipping the baseline. Teams sometimes pay for full updates when a prompt or LoRA run would match them.
• Wrong template or tokenizer. The same issue as in SFT. Format the data exactly as at inference.
• Data leakage. Near duplicates across train and test, or target information inside a serialized table row, inflate scores.
• Overtraining. Many epochs on a small set memorize it.
• Storing too many copies. Teams keep dozens of full variants and then cannot serve or audit them. Prefer adapters for many variants.
• Ignoring precision issues. Pure 16-bit optimizer states can cause drift. Keep master weights and statistics in higher precision.
• No resume plan. Without regular checkpoints, a failure at hour 30 costs the whole run.
• Mismatched missingness for tables. Training with random gaps and deploying on structured gaps makes imputation look better than it is.

Frequently asked questions

What is full fine-tuning?

Full fine-tuning is the practice of continuing to train a pre-trained model with all of its parameters trainable, so every weight can change. It produces a complete new copy of the model that is specialized for the new data or task, unlike methods that freeze the base and train a small add-on.

How is full fine-tuning different from LoRA?

Full fine-tuning updates every weight, while LoRA freezes the base model and trains small low-rank matrices that are added to chosen layers. LoRA needs far less memory and storage and carries less forgetting risk, while full fine-tuning has a higher quality ceiling when the new data is large and very different from the original training data.

How much memory does full fine-tuning need?

A rough rule is about 16 bytes per parameter for weights, gradients, master weights and two optimizer states in mixed precision, plus activations. A model with 7 billion parameters therefore needs on the order of 100 GB for states, which is why sharding, offloading or 8-bit optimizers are common.

When should I choose full fine-tuning over parameter-efficient methods?

Choose it when you have a large, high-quality dataset that is far from the model's original data, when a measured comparison shows that low-rank methods leave a real gap, and when you have the compute and storage to support it. For small data, quick experiments or many variants, parameter-efficient methods are usually the better choice.

Does full fine-tuning cause catastrophic forgetting?

It can, because all weights are free to move away from their pre-trained values. The risk grows with high learning rates, many epochs and data that differs from the original. You can reduce it with low rates, early stopping, replay of earlier data, weight regularization and by tracking a general benchmark during training.

Can I fully fine-tune a quantized model?

Not directly in the usual sense, because low-bit weights are not suited to small gradient updates. The common approach is QLoRA, which keeps the base in 4-bit and trains adapters on top. If you need full updates, train in 16-bit with 32-bit master weights and quantize the finished model afterward.

Is full fine-tuning useful for tabular foundation models?

It can be, when you hold a large labeled corpus from one sector and a stable schema, because it lets the whole network adapt to that sector's value ranges, missing-value patterns and rare categories. For most single-table tasks, in-context learning is simpler and avoids forgetting, so test it first.

How many examples do I need for full fine-tuning?

There is no fixed number, but full updates make most sense with thousands to millions of clean examples. With only dozens or hundreds, the model can memorize the set, and a low-rank method or few-shot prompting usually works as well at far lower cost.

What learning rate should I use?

Start low, often around 1e-5 for mid-size models and lower for larger ones, with a short warmup and a decaying schedule. Then tune on a validation set while watching a general benchmark. If the loss spikes or general ability drops, reduce the rate.

Do I need many GPUs for full fine-tuning?

Small models of a billion parameters or less can be trained on a single large accelerator with checkpointing and mixed precision. Larger models usually need several devices with sharding such as ZeRO or FSDP, or CPU offload, because the optimizer states alone can exceed the memory of one device.

Glossary

• AdamW. An optimizer that adapts step sizes per parameter and applies weight decay separately.
• Backpropagation. The method that computes gradients layer by layer.
• bf16. A 16-bit floating-point format with a wide range, common in training.
• Catastrophic forgetting. Loss of earlier skills after new training.
• Checkpoint. A saved snapshot of weights and optimizer state.
• Fine-tuning. Continuing training of a pre-trained model for a new purpose.
• Gradient accumulation. Summing gradients across steps before updating.
• Gradient checkpointing. Recomputing activations to save memory.
• Master weights. Higher-precision copies of weights kept by the optimizer.
• Offloading. Moving states to CPU memory or disk to save device memory.
• Optimizer state. Per-parameter statistics kept by the optimizer.
• Parameter-efficient fine-tuning. Methods that train only a small share of parameters.
• Replay. Mixing earlier data into new training.
• Sharding. Splitting model states across devices.
• Warmup. A short period of rising learning rate at the start.
• Weight decay. A penalty that discourages large weights.

Summary

Full fine-tuning updates every parameter of a pre-trained model on new data. It offers the highest adaptation capacity and no inference overhead, but it needs about sixteen bytes of training state per parameter, a full copy of the weights per variant, and careful control of the learning rate to avoid forgetting. Use it when you have large, clean, distant-domain data and a measured gap over cheaper options such as LoRA, QLoRA or DoRA.

For tabular foundation models, pre-training is itself full training, and further full updates can specialize the network to a sector's value ranges, missing-value patterns and rare categories for imputation, anomaly detection and sector classification. Because in-context learning already adapts these models to a new table without training, reserve full updates for stable schemas with large labeled corpora, protect earlier sectors with replay or adapters, and always compare against the cheaper baseline. More background is in /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: