QLoRA

Overview

QLoRA is a fine-tuning method that trains LoRA adapters on top of a base model stored in 4-bit precision. The base weights are frozen and quantized. The adapters are small, stay in higher precision, and receive all the gradients. During the forward and backward pass, the 4-bit weights are dequantized block by block into a compute type such as bfloat16, used for the matrix multiplication, and then discarded. The result is a large drop in memory with quality that stays close to 16-bit LoRA and often close to full fine-tuning.

The practical effect is that models that once needed several high-end accelerators to fine-tune can be tuned on one GPU. QLoRA rests on three ideas: a 4-bit data type called NormalFloat, double quantization of the scaling constants, and paged optimizers that spill optimizer state to host memory during memory spikes. It is supported by bitsandbytes and the PEFT ecosystem, and by tools such as Unsloth, Axolotl, and LLaMA-Factory.

What it is

QLoRA stands for quantized LoRA. It is not a new adapter design. It is a recipe that joins two known techniques: weight quantization to shrink the frozen model and low-rank adaptation to learn the task.

The one-sentence version

Store the frozen model in 4 bits, keep small adapters in 16 bits, and backpropagate through the dequantized weights into the adapters.

An everyday analogy

Picture a huge reference library you must carry on a trip. Full-precision books are heavy. You photograph every page at lower resolution and carry the compact copies. When you need a page, you enlarge it just for reading, then put it away. You also carry a small notebook where you write your own corrections. The compressed library is the 4-bit base. The notebook is the LoRA adapter.

What is frozen and what is trained

What QLoRA is not

• It is not post-training quantization for deployment. The aim is to make training possible, not to ship a 4-bit model, although you can do so.
• It is not a way to speed up training. Dequantization adds work, so QLoRA often runs slower per step than 16-bit LoRA on the same hardware. It trades speed for memory.
• It does not train the 4-bit weights. They never change.

Where it came from

The memory problem

By the early 2020s, language models had grown so large that full fine-tuning needed far more memory than a single device offers. Training stores weights, gradients, and optimizer states. With Adam in mixed precision, the total can reach many times the size of the weights.

Building blocks that already existed

• LoRA, 2021, cut gradient and optimizer memory by training small matrices.
• 8-bit inference and optimizers, from the bitsandbytes library, showed that low-bit storage of large matrices can keep quality. Blockwise quantization was a key technique.
• Post-training quantization methods such as GPTQ and AWQ showed that 4-bit weights can serve models well, although they were aimed at inference.
• Gradient checkpointing and mixed precision were standard memory tools. See gradient checkpointing and mixed precision.

The QLoRA paper

The paper "QLoRA: Efficient Finetuning of Quantized LLMs" appeared in 2023. It introduced the NormalFloat 4-bit data type, double quantization, and paged optimizers, and it showed that backpropagating through a frozen 4-bit model into LoRA adapters could match 16-bit fine-tuning quality on the studied benchmarks. It also reported that data quality mattered more than data size for instruction tuning.

After the paper

QLoRA quickly became the standard way to tune large open models on modest hardware. Later variants explored better initialization for quantized bases, different bit widths, and combinations with DoRA and rsLoRA.

How it works

Step 1: quantize the base weights to 4 bits

Weights are split into small blocks, for example 64 values each. Each block is scaled by its largest absolute value so values fall in a fixed range. Each value is then mapped to one of 16 levels. The 16 levels are not evenly spaced. They follow the NormalFloat design.

Why NormalFloat

Trained neural network weights are roughly bell-shaped, centered near zero. An evenly spaced grid wastes levels on the rare large values. NormalFloat places the 16 levels at quantiles of a normal distribution, so each level is used about equally often. Using levels well means less information is lost for the same four bits.

An analogy: if you are marking heights of people on a ruler with only 16 ticks, you put more ticks near the common heights and fewer at the extremes.

Step 2: store the scale per block

Each block needs one scaling constant. With 64 values per block and a 32-bit constant, that is half a bit per weight of overhead. It looks small but adds up on billions of weights.

Step 3: double quantization

The scaling constants themselves are quantized, for example to 8 bits in groups, with their own second-level constants. This lowers the overhead per weight further. It is a small gain per weight, but meaningful at scale.

Step 4: attach LoRA adapters

Adapters A and B are added to the chosen layers, as in LoRA. They are stored in 16-bit and trained. The recipe commonly adapts all linear layers, since adapting only a few attention layers was found to leave quality on the table with a quantized base.

Step 5: forward pass with on-the-fly dequantization

For each layer, the 4-bit block is dequantized into the compute type, multiplied with the input, and the result is added to the adapter path:

y = dequant(W_4bit) x + (alpha / r) * B (A x)

The dequantized weight exists only briefly. Memory stays low because the full 16-bit weight is never stored for the whole model at once.

Step 6: backward pass

Gradients flow through the dequantized computation back to the adapter matrices. No gradient is computed for the frozen 4-bit weights. Activations are still needed, so gradient checkpointing is common.

Step 7: paged optimizers

Long sequences can cause sudden memory spikes. Paged optimizers keep optimizer state in unified memory so that, when the device runs out, pages move to host memory and return when needed. This prevents a crash on a single bad batch. The idea is the same as virtual memory in an operating system.

Step 8: save and use

You save only the adapter. At inference you have options.

• Load the 4-bit base and the adapter together. This is memory-light.
• Dequantize the base to 16-bit, merge the adapter, and serve a full-precision model.
• Merge and then re-quantize with a post-training method for deployment. Quality can shift slightly, so evaluate.

The memory arithmetic

A model with seven billion parameters needs about 14 gigabytes in 16-bit. At 4 bits it needs about 3.5 gigabytes plus small overhead. For 65 billion parameters the figures are about 130 gigabytes versus about 33 gigabytes. The adapters and their optimizer state add a small amount, and activations add more depending on batch and sequence length.

These are weight figures only. Real usage includes activations, adapter state, and framework overhead.

Key concepts and terms

A worked example

A small numeric walk-through of quantization

Take a block of four weights: [0.10, -0.40, 0.05, 0.80].

1. The largest absolute value is 0.80. Divide every weight by 0.80 to get [0.125, -0.5, 0.0625, 1.0].
2. Each normalized value is rounded to the nearest of 16 NF4 levels. The exact levels are fixed constants of the data type. Suppose the nearest levels are 0.1, -0.52, 0.08, and 1.0.
3. The block stores four 4-bit indices plus the constant 0.80.
4. On dequantization, multiply levels by 0.80 to get about [0.08, -0.42, 0.06, 0.80].

The small differences are the quantization error. QLoRA works because the adapter can learn to compensate for such errors and for the task shift at once.

A code example with common libraries

import torch
from transformers import (
    AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model_name = "example-org/large-base-model"  # placeholder

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, quantization_config=bnb, device_map="auto"
)

model = prepare_model_for_kbit_training(model)  # casts norms, enables checkpointing

lora = LoraConfig(
    r=64,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules="all-linear",
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()

Training arguments

args = TrainingArguments(
    output_dir="out/qlora-a",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    num_train_epochs=1,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    optim="paged_adamw_8bit",   # paged optimizer
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    save_strategy="steps",
    save_steps=200,
)

Reading the memory profile

Imagine a single 24 GB GPU and a 13 billion parameter base. Weights in 4-bit take about 7 GB. Adapters with rank 64 over all linear layers take a few hundred megabytes with state. The rest of the memory is activations, which you control with batch size, sequence length, and checkpointing. If an out-of-memory error appears, reduce the micro-batch size and raise accumulation steps, which keeps the effective batch size the same.

Merging after QLoRA

from peft import PeftModel

base16 = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, device_map="cpu"
)
tuned = PeftModel.from_pretrained(base16, "out/qlora-a")
merged = tuned.merge_and_unload()
merged.save_pretrained("out/merged-16bit")

Merging into the 16-bit original is usually preferred to merging into a dequantized copy, because the adapter was trained against the quantized weights. The small mismatch between the two is generally tolerable, but you should test the merged model.

Strengths and limitations

Strengths

• Large reduction in memory for the frozen base.
• Quality close to 16-bit LoRA on many reported tasks.
• Makes very large models reachable on a single GPU.
• Paged optimizers guard against memory spikes.
• Small adapter files, as with LoRA.
• Works with standard training stacks and many wrappers.
• Lets practitioners iterate on large models without a cluster.

Limitations

• Slower than 16-bit LoRA per step because of dequantization.
• Quantization error is baked into the frozen weights. The adapter can offset some of it, not all.
• Outlier-heavy models can lose more quality at 4 bits.
• Merging back into low-bit weights can shift behavior.
• Some architectures or custom layers lack quantized kernels.
• Activation memory is not reduced, so long sequences still strain memory.
• Hardware and kernel support vary, so exact speed and memory differ by setup.
• Reproducibility can be harder because of non-deterministic kernels.

Where QLoRA tends to shine

• Instruction tuning and chat tuning of large open models.
• Domain adaptation with a few thousand to a few hundred thousand examples.
• Rapid experiments where hardware is limited.

Where to be careful

• Tasks that need exact numeric precision from the base weights.
• Very small base models, where 16-bit LoRA already fits and quantization adds error for little gain.
• Production serving paths that cannot use the same quantization kernels.

How it compares with alternatives

Quantization methods compared

Choosing between them

• If the model fits in 16-bit with room for activations, plain LoRA is simpler and faster.
• If it does not fit, QLoRA is the standard answer.
• If you want to train every weight but the optimizer is too large, look at GaLore.
• If you must serve at low latency, quantize for serving after training with a method built for inference.

Role in tabular foundation models

Tabular foundation models learn from raw tables. Inputs have rows and columns, mixed types, missing cells, and frequently no column names. They are pre-trained on large collections of synthetic or real tables, solve tasks through in-context learning, and may be updated over time. See /model/architecture and /model/research. QLoRA touches these models mainly through scale and cost.

Why scale matters for tables

Early tabular foundation models were modest in size. As they grow toward billions of parameters, memory for fine-tuning becomes a real constraint. The same trade that helped language models applies: quantize the frozen base, train small adapters. A model of more than a billion parameters can be adapted per customer on a single GPU rather than on a cluster.

Adapting to a new schema

A pre-trained table model has seen many schemas. A new one has particular numeric ranges, categories, and missing-value habits. A QLoRA adapter learns those habits while the base stays compact. You keep one quantized base on disk and many small adapters, one per schema family. See TabPFN, TabICL, and TabSTAR for models of this kind.

Interaction with in-context learning

In-context learning conditions on labeled rows given at run time. Quantizing the base changes its numeric behavior slightly, which can affect how well it reads these rows. When you run in-context inference on a 4-bit base, validate on tasks with the same row counts and feature counts that you will see in production. An adapter trained with the quantized base in the loop adapts to those small numeric shifts, which is an advantage over quantizing after training.

Mixed types, missing values, and no column names

Numeric precision matters for tables. Value encoders often take raw numbers, and small perturbations can change how a cell is embedded. For this reason, keep the input embedding layers and normalization layers in higher precision, as the standard QLoRA preparation step does for norms, and quantize the large internal linear layers. This keeps the sensitive part, where raw cell values enter the network, accurate. Missing-value indicators and column-type embeddings are tiny, so there is no memory reason to quantize them.

Continual learning

Each new data source or time period can get its own adapter over the same quantized base. The base never changes, so earlier adapters stay valid. This gives a low-cost form of continual learning that avoids catastrophic forgetting, in contrast to methods that regularize weights such as EWC. See also continued pre-training for the heavier alternative.

Imputation, anomaly detection, and sector classification

• Imputation: Train an adapter on masked-cell objectives over the target table so that predictions of missing values reflect its true missingness pattern. See masked cell modeling and imputation networks.
• Anomaly detection: Train an adapter on normal rows of one asset class so that unusual rows score high. See anomaly detection.
• Sector and semantic classification: Tune the encoder and a small head toward a new taxonomy of industries or column meanings. See semantic sector heads.

Concrete mapping

Honest caveats

• Many tabular models are small enough that 16-bit LoRA or even full fine-tuning fits. QLoRA is most useful when the model is large.
• Table models differ from language models in layer names and in embedding design, so check which layers can be quantized by your library.
• Always compare the quantized, adapted model with a 16-bit reference on held-out tables. See /model/benchmarks.

Practical guidance and best practices

Quantization settings

• Use NF4 with double quantization as the default.
• Use bfloat16 as the compute type when the hardware supports it, otherwise float16.
• Keep normalization and embedding layers in higher precision.

Adapter settings

• Target all linear layers.
• Start with rank 16 to 64 and tune on validation data.
• Keep alpha in a reasonable ratio with rank. See LoRA and rsLoRA.
• Add light dropout for small datasets.

Optimizer and schedule

• Use a paged 8-bit or 32-bit AdamW variant.
• Start near a learning rate of 2e-4 for small models and lower it for very large ones.
• Use warmup and cosine or linear decay.
• Use gradient clipping to keep updates stable.

Memory management

• Lower the micro-batch size first, then raise gradient accumulation.
• Turn on gradient checkpointing. See gradient checkpointing.
• Cap the sequence length to what the task needs.
• Pack short examples to use compute well.

Evaluation

• Evaluate the adapter against the 16-bit base with the same prompts.
• Evaluate after merging, since the merged model is what you may deploy.
• Track a general-ability set to detect drift.

Deployment

• For small teams, serving the 4-bit base plus adapter is simplest.
• For high throughput, merge and use a serving engine built for the target format. See vLLM.
• If you re-quantize after merging, check quality again.

Tooling

• bitsandbytes provides the 4-bit kernels.
• PEFT provides the adapters.
• TRL and SFT recipes provide training loops.
• Unsloth offers optimized kernels for the same workflow.

Troubleshooting guide

Deciding whether 4 bits is worth it

Ask three questions before you start.

• Does the 16-bit base fit on your device with room for activations and adapter state? If yes, plain LoRA is simpler.
• Will the final model be served in 4-bit as well? If yes, training against the same quantized weights helps, because the adapter adapts to that exact error.
• Is the task sensitive to small numeric changes? If yes, run a short comparison against a 16-bit adapter before committing.

Data tips specific to quantized training

• Quantization noise acts like a small regularizer, but it does not replace clean data.
• Deduplicate the training set. See deduplication.
• Keep a fixed validation set across all runs so changes in loss are comparable.

A small experiment plan

1. Train a short run in 16-bit LoRA on a subset, if memory allows, to get a reference loss.
2. Train the same subset with QLoRA using identical data order and seed.
3. Compare validation loss and a few task metrics.
4. If the gap is small, scale up the QLoRA run to the full data.
5. If the gap is large, raise the rank, add target modules, or try a larger bit width for sensitive layers.

Common mistakes

• Expecting QLoRA to be faster. It saves memory, not time.
• Using FP4 without a reason when NF4 is the better-tested choice.
• Leaving the compute type at float32 and wasting memory and time.
• Skipping the preparation step that casts norms and enables input gradients for checkpointing.
• Targeting only a few attention layers and then seeing weak results.
• Merging adapters into a quantized copy and shipping without evaluation.
• Using a learning rate tuned for full fine-tuning.
• Ignoring padding side and chat template, then blaming quantization for odd outputs.
• Comparing to a 16-bit baseline that used a different data order or seed.
• Training on very long sequences without checkpointing.
• Assuming 4-bit is always safe for tasks that need exact numeric behavior.
• Forgetting to store the base model identifier, quantization settings, and tokenizer version alongside the adapter.

Frequently asked questions

What is QLoRA in simple terms?

QLoRA is LoRA fine-tuning applied to a base model that has been compressed to 4 bits. The compressed weights are frozen, small 16-bit adapters are trained, and the weights are expanded on the fly during each calculation. This keeps memory low while preserving most of the quality of 16-bit training.

How much memory does QLoRA save?

The frozen weights take about a quarter of their 16-bit size, plus a small overhead for scaling constants. Total savings also depend on activations, batch size, and sequence length, which QLoRA does not shrink. In practice, models several times larger can be tuned on the same GPU than with 16-bit LoRA.

What is NF4?

NF4 stands for NormalFloat 4-bit. It is a 4-bit data type whose 16 levels are placed at quantiles of a normal distribution, which matches the bell-shaped distribution of trained weights. Using levels in proportion to where weights actually lie reduces error compared with evenly spaced levels.

What is double quantization?

Block-wise quantization needs one scaling constant per block of weights. Double quantization quantizes those constants as well, using a second small set of constants. It shaves a fraction of a bit per weight, which is meaningful across billions of weights.

What are paged optimizers?

Paged optimizers keep optimizer state in memory that can move between the GPU and host memory automatically. When a batch causes a memory spike, pages are evicted to host memory instead of crashing the run. This is similar to how an operating system pages memory to disk.

Is QLoRA as accurate as regular LoRA?

On the benchmarks reported in the original work, it matched 16-bit methods closely. Results vary with model, data, and rank, so you should measure on your own validation set. Quantization error is real, and the adapter can only partly offset it.

Can I merge a QLoRA adapter into the base model?

Yes. The safest route is to load the base in 16-bit, apply the adapter, and merge there. Merging into a dequantized copy of the 4-bit weights is possible but can introduce small differences. Always evaluate the merged model before deployment.

Does QLoRA work for models other than language models?

It works for any model with large linear layers that a library can quantize, including vision and speech models. For tabular foundation models it can help when the model is large, though many table models are small enough that plain LoRA is enough.

Why is QLoRA slower than LoRA?

Each forward and backward pass must dequantize the 4-bit blocks into the compute type before the matrix multiplication. That extra work costs time. Optimized kernels reduce it, but the method still trades some speed for memory.

Should I use QLoRA or full fine-tuning?

Use full fine-tuning when you have the memory and the task needs the highest possible quality. Use QLoRA when the model does not fit otherwise or when you need many cheap, separate task adapters. Many teams prototype with QLoRA and move to larger runs only if the results demand it.

Glossary

• 4-bit quantization: Storing each weight with one of 16 levels.
• Adapter: Small trainable matrices added to a frozen model.
• Block size: The number of weights that share one scaling constant.
• bfloat16: A 16-bit floating-point type with a wide exponent range.
• Compute dtype: The numeric type used during the matrix multiplication.
• Dequantization: Converting low-bit values back to a higher-precision type.
• Double quantization: Quantizing the scaling constants.
• Gradient checkpointing: Recomputing activations in the backward pass to save memory.
• NF4: NormalFloat 4-bit data type.
• Outlier: A value much larger than typical, which is hard for low-bit schemes.
• Paged optimizer: Optimizer state that can move to host memory on demand.
• Post-training quantization: Compressing a trained model without further training.
• Rank: The inner dimension of the LoRA matrices.
• Scaling constant: The per-block value used to map weights into the quantization range.

Summary

QLoRA combines a frozen 4-bit base model with trainable LoRA adapters. NormalFloat quantization, double quantization of scaling constants, and paged optimizers together cut memory enough to fine-tune very large models on one GPU, with quality that stays near 16-bit LoRA on many tasks. The price is slower steps, some quantization error, and extra care when merging and deploying.

For tabular foundation models, QLoRA is most useful as models grow large. It supports cheap per-schema adapters, adaptation to shifted distributions, tuned imputation and anomaly baselines, sector classification, and continual learning without forgetting, all over one compact shared base. Keep sensitive input layers in higher precision, validate against a 16-bit reference, and see the related pages on LoRA, DoRA, GaLore, bitsandbytes, and the benchmarks.

Related

It belongs to Fine-tuning in my stack, under LLMs, retrieval, and evaluation. Others in the same group: