bitsandbytes

Overview

bitsandbytes is an open source library that lets PyTorch models use 8-bit and 4-bit numbers instead of 16-bit or 32-bit numbers. It supplies quantized linear layers, 8-bit optimizers and paged optimizers. With it, a model that once needed several large GPUs can often be loaded and fine-tuned on a single card.

The library is best known as the engine behind QLoRA, where a frozen 4-bit base model is trained through small LoRA adapters. It also works for plain inference, where it shrinks memory use with only a small loss in quality. Because it plugs into the Transformers ecosystem through a single config object, many people use it without ever calling it directly.

What it is

bitsandbytes is a Python package with a compiled core written for GPU execution. It wraps low-level kernels that quantize, dequantize and multiply low-precision data. It exposes them as ordinary PyTorch modules, so a normal nn.Linear layer can be swapped for a quantized one.

Think of it as a set of drop-in parts. You keep your model design. You replace the heaviest parts, the large weight matrices and the optimizer state, with compact versions.

The library has three main families of features.

• Quantized linear layers for 8-bit (the LLM.int8 method) and 4-bit (FP4 and NF4 formats).
• 8-bit optimizers such as Adam, AdamW, Lion and SGD variants, which store their running statistics in 8 bits.
• Paged optimizers, which move optimizer state between GPU and CPU memory when the GPU runs short.

What it does not do

It is not a model format, a serving engine or a training framework. It does not define how a model is stored on disk the way GGUF does. It does not batch requests like vLLM. It is a numerical building block that other tools call.

Where it runs

The original and most mature backend targets NVIDIA GPUs through CUDA. Support for other hardware has grown over time, but features differ by backend. Always read the current documentation for your device before planning a project around it.

Where it came from

Two research lines fed the library. The first was work on 8-bit optimizers, published in 2021 as "8-bit Optimizers via Block-wise Quantization". It showed that the running averages kept by optimizers like Adam can be stored in 8 bits without hurting training results. This removes a large share of training memory, because Adam keeps two extra numbers per parameter.

The second was "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale", published in 2022. It found that large language models contain a few feature dimensions with huge values, called outliers. Naive 8-bit rounding ruined quality at larger scales because of these outliers. The paper's fix was to keep the outlier dimensions in 16-bit and quantize everything else to 8-bit.

The third step was "QLoRA: Efficient Finetuning of Quantized LLMs", published in 2023. It introduced the NF4 data type, double quantization and paged optimizers, all implemented in bitsandbytes. It showed that a 4-bit frozen base model plus trainable adapters can match 16-bit fine-tuning quality on many tasks.

After that, integration into Hugging Face libraries made the feature one flag away: load_in_8bit or load_in_4bit, later wrapped in BitsAndBytesConfig. That ease of use explains much of its spread.

How it works

The simplest way to see quantization is to think about a ruler. A 16-bit number is like a ruler with very fine marks. An 8-bit number is a ruler with 256 marks. A 4-bit number is a ruler with only 16 marks. When you measure something you must round to the nearest mark. Fewer marks mean more rounding error, but also far less storage.

Step 1: split the weights into blocks

Weights are not rounded as one giant list. They are cut into small blocks, for example 64 values at a time. Each block gets its own scale. This is like giving each room in a house its own thermostat instead of one setting for the whole building. A single huge value then only harms its own block.

Step 2: find the scale

For each block, the library finds the largest absolute value. That value, called the absmax, becomes the scale. Every value in the block is divided by it, so all values fall between -1 and 1.

Step 3: round to the nearest code

After scaling, each value is mapped to the nearest entry in a small table of allowed values, called a code book. For 8-bit integer quantization the table is evenly spaced. For 4-bit NF4 the table is spaced to match a bell curve.

Step 4: store compactly

The rounded indices are packed. For 4-bit data, two values share one byte. The scales are stored beside them. The original 16-bit weights are discarded.

Step 5: dequantize on the fly

Matrix multiplication hardware works best on 16-bit numbers. So at run time, the library reads a block, expands it back to 16-bit using the scale and code book, and multiplies. The expanded copy is temporary. Only the compact version lives in memory. You trade a little speed for a large memory saving.

How LLM.int8 handles outliers

In 8-bit mode the method uses vector-wise quantization, with a separate scale for each row of the input and each column of the weights. It then looks at the input features. Columns whose magnitude exceeds a threshold, commonly set near 6.0, are treated as outliers. Those columns are multiplied in 16-bit. The rest are multiplied in int8. The two results are added.

Picture a delivery truck. Most packages are small and go in standard boxes. A few huge items would break the box system, so they travel separately. The total still arrives together.

How NF4 works

Neural network weights are usually spread like a bell curve around zero. NF4, short for 4-bit NormalFloat, places its 16 code values so that each covers an equal share of that bell curve. Common small values get finer marks. Rare large values get coarser ones. This is better than evenly spaced marks when the data is bell shaped.

Double quantization

Each block stores a scale, and those scales also take space. Double quantization quantizes the scales themselves. It is a savings on the savings, small per block but meaningful across billions of weights.

8-bit optimizers

During training, Adam keeps a first and second moment for each parameter. In 32-bit that is eight extra bytes per parameter. The 8-bit version stores them in blocks with dynamic quantization and dequantizes them for the update step. The weights themselves can stay in higher precision. Training behaves much like the full version while the optimizer state shrinks to about a quarter.

Paged optimizers

Memory spikes happen, for example with a long sequence in a batch. Paged optimizers use unified memory so that optimizer state can spill to CPU RAM when the GPU is full, and come back when needed. It works like a desk with an overflow shelf. You do not crash when the desk is full. You just walk to the shelf for a moment.

Key concepts and terms

A worked example

Numeric walk-through

Take a block of four weights: 0.10, -0.40, 0.25, 0.80. The absmax is 0.80. Dividing by it gives 0.125, -0.5, 0.3125, 1.0.

Now use a simple 8-bit integer scheme with range -127 to 127. Multiply each scaled value by 127 and round.

• 0.125 times 127 is 15.875, which rounds to 16.
• -0.5 times 127 is -63.5, which rounds to -64 (rounding away from zero here).
• 0.3125 times 127 is 39.69, which rounds to 40.
• 1.0 times 127 is 127.

To dequantize, divide by 127 and multiply by 0.80. The value 16 becomes about 0.1008. The value -64 becomes about -0.4031. The errors are around 0.001 and 0.003, small compared with the values. In 4-bit mode there are only 16 levels, so the errors are larger, which is why NF4 spaces its levels carefully.

Loading a model in 4-bit

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model_id = "example-org/example-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)
print(model.get_memory_footprint() / 1e9, "GB")

The config says: store weights as NF4, quantize the scales again, and do the math in bfloat16. The device_map option places layers on available devices.

Loading in 8-bit

from transformers import BitsAndBytesConfig

config_8bit = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=config_8bit,
    device_map="auto",
)

The threshold controls which columns count as outliers and stay in 16-bit.

Using an 8-bit optimizer

import bitsandbytes as bnb
import torch.nn as nn

model = nn.Sequential(nn.Linear(1024, 4096), nn.ReLU(), nn.Linear(4096, 10))
optimizer = bnb.optim.AdamW8bit(model.parameters(), lr=1e-4, weight_decay=0.01)

# paged variant for large jobs
paged = bnb.optim.PagedAdamW8bit(model.parameters(), lr=1e-4)

The call pattern matches a normal PyTorch optimizer, so training loops need no change.

Fine-tuning with adapters on a 4-bit model

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)
lora = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()

Only the adapter weights train. The 4-bit base stays frozen. This is the QLoRA recipe, and PEFT provides the adapter side.

Memory arithmetic

A model with 7 billion parameters needs about 14 GB in 16-bit, since each parameter takes 2 bytes. In 8-bit it needs about 7 GB. In 4-bit it needs about 3.5 GB, plus a small overhead for scales. Activations, the key-value cache and adapter state are extra. These are rough numbers for planning, not guarantees.

Strengths and limitations

Strengths

• Large memory savings with simple code changes.
• Tight integration with Transformers, PEFT and the wider PyTorch stack.
• Works for both inference and training.
• NF4 and double quantization give good quality at 4 bits for fine-tuning.
• 8-bit optimizers cut training memory without changing the training loop.
• No calibration dataset is needed. Quantization happens at load time.

Limitations

• Speed is not always better. Dequantization costs time, so latency can be higher than 16-bit, especially at small batch sizes.
• Quantization at load time repeats on every start, unless you save the quantized weights.
• Quality can drop on tasks that need exact reasoning or precise numbers.
• Hardware support differs between backends and versions.
• It targets PyTorch only. It is not a general format that other runtimes can read.
• Post-training methods like GPTQ and AWQ often give faster inference kernels at 4 bits, because they pack weights for fused kernels.
• Training the quantized base weights directly is not the goal. Gradients flow through them to the adapters.

How it compares with alternatives

A short rule of thumb: pick bitsandbytes when you want to fine-tune or experiment with minimal setup. Pick GPTQ, AWQ or ExLlamaV2 when you mainly serve a fixed model fast. Pick GGUF when you need to run on varied hardware.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing values, and sometimes no useful column names. Models like TabPFN are pre-trained on large numbers of synthetic tables, see synthetic data generation, and then predict on new tables through in-context learning. The training cost is where bitsandbytes helps.

Fitting a large table model on limited hardware

A tabular foundation model keeps many parameters because it must learn general rules about data, not one dataset. Pre-training or adapting such a model needs memory for weights, gradients and optimizer state. An 8-bit optimizer cuts the optimizer share. Loading a frozen backbone in 8-bit or 4-bit cuts the weight share. This lets teams adapt a large table model on one or a few GPUs.

Adapter-based continual learning

Continual learning means updating a model with new tasks without erasing old skills. A frozen quantized base plus small adapters is a natural fit. Each new domain gets its own adapter, and the base stays untouched. This avoids forgetting by design. It is the same idea as QLoRA, applied to a model that reads rows and columns instead of text. Adapters can be added for a new industry, a new schema style or a new kind of missingness.

Models that read tables as text

Some systems serialize a row as text and feed it to a language model. Cells like "age: 41, plan: gold, balance: missing" become tokens. Here bitsandbytes is used just as in any language model. A 4-bit base with adapters can be tuned for imputation, where the model fills in a missing cell, and for anomaly detection, where it flags a row that looks unlikely. See imputation networks and anomaly detection.

Semantic and sector classification

Another table task is deciding what a column means, or what industry a dataset belongs to, even when column names are absent or cryptic. A language-model head on a quantized backbone can classify columns by their values: the shape of the numbers, the pattern of the strings, the share of blanks. Low-precision loading keeps such classifiers cheap to host. See semantic sector heads.

Precision concerns for numeric data

Tables carry exact numbers: money, counts, sensor readings. Rounding errors in weights are not the same as rounding errors in data, but a heavily quantized model may become less sharp on fine numeric distinctions. Test any quantized tabular model against its full-precision version on held-out tables, and compare calibration of predicted probabilities, not just accuracy. The input encoders and the output heads are small and sensitive, so many setups keep them in higher precision and quantize only the big middle blocks.

Research context

For the design of the model family, see model architecture. For measured results and comparison methods, see benchmarks and research notes.

Practical guidance and best practices

Start with the right data type

• Use NF4 for fine-tuning a 4-bit base. It is the default choice for good reason.
• Use bfloat16 as the compute dtype if your GPU supports it. It avoids many overflow problems of float16.
• Turn on double quantization when memory is tight.

Measure before and after

1. Record memory use and latency of the 16-bit model.
2. Load in 8-bit and measure again.
3. Load in 4-bit and measure again.
4. Run your own evaluation set on each version.

Do not trust a general claim that quantization is free. Check on your task.

Keep sensitive layers in higher precision

The output head and the embedding layer are often skipped. The llm_int8_skip_modules option lets you list modules that stay in higher precision. This is a cheap way to protect quality.

Prepare the model for training

Call prepare_model_for_kbit_training before adding adapters. It handles details like casting normalization layers to a stable type and enabling gradient checkpointing hooks.

Use gradient checkpointing

Activations can dominate memory once weights are small. Gradient checkpointing trades extra compute for lower activation memory. Combine it with 4-bit loading for long sequences.

Use paged optimizers for long runs

If a job sometimes hits out-of-memory errors on a few batches, a paged optimizer smooths the spikes.

Save the adapters, not the quantized base

Store the small adapter files. Reload the base in 4-bit at inference. If you merge adapters into a quantized base, remember the merge may require dequantizing first, which can add small errors.

Pin versions

The library, the Transformers package, PEFT and the GPU driver need to agree. Pin versions in a lock file and test upgrades on a copy of the job.

Example training config

model:
  name: example-org/example-model
  quantization:
    bits: 4
    type: nf4
    double_quant: true
    compute_dtype: bfloat16
adapter:
  method: lora
  rank: 16
  alpha: 32
  dropout: 0.05
optimizer:
  name: paged_adamw_8bit
  lr: 0.0002
training:
  gradient_checkpointing: true
  batch_size: 4
  grad_accum_steps: 8

Deploying after training

For serving at scale, consider converting the final model to a format built for inference, such as AWQ or GPTQ weights run by vLLM. The bitsandbytes path is excellent for training and prototyping, and inference engines often have faster paths for fixed models.

A simple memory budget

Plan memory before you start. A rough budget has four parts: the weights, the gradients, the optimizer state and the activations.

• Weights: at 16 bits each value takes 2 bytes. At 8 bits it takes 1 byte. At 4 bits it takes half a byte.
• Gradients: with adapter training, only the small adapter matrices have gradients, so this part is tiny.
• Optimizer state: Adam keeps two extra numbers per trained value. Eight-bit optimizers shrink that part by about three quarters compared with 32-bit state.
• Activations: these grow with batch size and sequence length. Gradient checkpointing trades extra compute for lower activation memory.

An everyday comparison helps. Think of packing for a trip. The weights are your clothes, the optimizer state is your toiletries, and the activations are the souvenirs you pick up on the way. Quantization vacuum-packs the clothes. It does not shrink the souvenirs, so a long sequence can still overflow the bag.

Choosing between 8-bit and 4-bit

Eight-bit loading is the safer first step. It keeps quality very close to the original model and needs no format choice. Four-bit loading saves roughly twice as much again, and it is the right choice when memory is the real limit. If you have room for 8-bit, you do not need 4-bit. If an 8-bit model already fits with room for activations, the extra savings may not be worth the small quality cost.

A short decision path:

1. Does the 16-bit model fit with room for training? Then skip quantization.
2. Does the 8-bit model fit? Use 8-bit and compare quality.
3. Still too large? Use 4-bit NF4 with double quantization and bfloat16 compute.
4. Still too large? Reduce sequence length, batch size or model size before pushing to lower precision.

Checking that quantization did not hurt

Run the same small evaluation set on the original and the quantized model. Compare task metrics, not only loss. For tabular work, compare predictions on held-out rows and look at calibration, since a model can keep its accuracy while its probabilities drift. Repeat the check after merging adapters, because merging adds another rounding step.

Debugging checklist

• The model loads but training loss is flat: check that adapters are attached and that the learning rate is not tiny.
• Loss becomes not-a-number: switch to bfloat16 compute, lower the learning rate and check for extreme input values.
• Out of memory despite 4-bit loading: reduce sequence length, turn on checkpointing, or use a paged optimizer.
• Import error about GPU kernels: confirm that the library build matches the installed CUDA runtime and driver.
• Results differ between machines: expect small numerical differences and compare with a tolerance.
• Slow generation: batch more requests, or export the merged model to a serving engine such as vLLM.

Working with other tools

bitsandbytes pairs well with PEFT for adapter management and with Axolotl, Unsloth or LLaMA-Factory for ready-made training recipes. For inference-only needs, calibrated methods such as GPTQ or AWQ may serve faster. For local use on laptops, convert to GGUF and run through llama.cpp. Memory tricks stack: mixed precision, gradient checkpointing and FSDP all combine with low-precision loading when versions allow.

Reproducibility notes

Fix random seeds for data order and adapter initialization. Record the library version, the CUDA version, the GPU model and the quantization config in your run log. Store the config next to the adapter files so that someone else can rebuild the exact setup later. Low-precision kernels are not guaranteed to be bit-identical across hardware, so judge reproducibility by metric spread over a few seeds rather than by identical numbers.

Tuning the adapter side

When the base is frozen in 4-bit, quality depends on the adapters. Three settings matter most.

• Rank: a higher rank gives adapters more capacity and uses more memory. Start small, such as 8 or 16, and raise it only if validation scores stall.
• Target modules: attaching adapters to all linear layers usually helps more than attaching them to attention projections alone.
• Scaling: the adapter output is multiplied by a scale factor. Changing rank and scale together can make runs hard to compare, so change one at a time. Variants such as rsLoRA adjust the scaling rule for higher ranks.

Data and batching tips

• Sort or bucket examples by length so that padding wastes less memory.
• Use gradient accumulation to reach a large effective batch size without extra activation memory.
• Keep a small validation split that you never train on, and evaluate it on a fixed schedule.
• Stop early when validation quality stops improving. Low-precision training can overfit small datasets quickly.

Monitoring a run

Watch four signals during training: training loss, validation loss, GPU memory use and step time. A healthy run shows a smooth falling loss, stable memory after the first few steps and steady step time. A rising memory curve points to a leak, often from keeping tensors in a Python list. Sudden spikes in loss often come from a bad batch or an unstable learning rate.

When not to use it

• The model already fits comfortably in 16-bit and speed is the goal.
• The target hardware has no supported GPU backend for your build.
• You need bit-exact outputs for audit reasons.
• The model is small, so savings are minor and quality loss is proportionally larger.

In these cases, plain mixed precision training or a calibrated inference method is simpler.

Security and supply chain notes

Download model weights only from sources you trust. Prefer safe tensor formats over pickled files, since pickled files can run code when loaded. Pin package versions in a lock file, and build container images from known bases. These steps matter more when low-level compiled GPU code is part of the stack.

Quick reference: common config fields

Treat these as a starting point. Validate each change on your own data before you keep it.

Common mistakes

• Assuming 4-bit loading makes inference faster. It makes it smaller. Speed depends on batch size and kernels.
• Forgetting to set the compute dtype, which falls back to a slower or less stable type.
• Training with a learning rate tuned for full fine-tuning. Adapter training usually wants a higher rate.
• Skipping evaluation on the target task and trusting a generic leaderboard.
• Quantizing a model that is already tiny. The savings are small and the quality loss is relatively larger.
• Mixing library versions and getting silent fallbacks or errors about missing GPU kernels.
• Merging adapters into a quantized model without checking the output afterward.
• Expecting identical results between runs on different GPUs. Low-precision kernels can differ slightly.
• Quantizing the embedding and output layers without testing, then blaming the method for poor text or poor class probabilities.
• Using 8-bit optimizers on parameters with very few values. Small tensors are normally left in 32-bit by design, and forcing them can hurt stability.

Frequently asked questions

What is bitsandbytes used for?

bitsandbytes is used to cut the memory needed by large neural networks in PyTorch. It provides 8-bit and 4-bit quantized layers for loading models, 8-bit optimizers for training, and paged optimizers that spill to CPU memory. Its best-known use is QLoRA fine-tuning, where a frozen 4-bit base model is trained through small adapters.

What is the difference between LLM.int8 and NF4?

LLM.int8 is an 8-bit method that keeps rare outlier feature columns in 16-bit and multiplies the rest in int8. NF4 is a 4-bit data type whose 16 values are placed to fit bell-shaped weights. The first is mostly used to shrink inference memory with little quality loss. The second is mostly used as the storage format for QLoRA.

Does 4-bit quantization hurt model quality?

Usually a little, and sometimes not enough to notice. The loss depends on model size, task and which layers are quantized. Larger models tend to tolerate it better. You should evaluate on your own data, and compare against the 16-bit model, before relying on the result.

Is bitsandbytes faster than 16-bit inference?

Not by default. The weights must be expanded to 16-bit during the math, which costs time. At small batch sizes, memory transfer savings can help, but dedicated kernels from tools like GPTQ, AWQ or ExLlamaV2 are often faster. Choose bitsandbytes for convenience and memory, not for peak speed.

How does QLoRA relate to bitsandbytes?

QLoRA is a training recipe, and bitsandbytes is the library that makes its memory trick possible. The recipe loads the base model in 4-bit NF4 with double quantization, freezes it, and trains LoRA adapters in higher precision, with a paged optimizer for memory spikes. The adapters come from PEFT and the quantized layers come from bitsandbytes.

Do I need calibration data?

No. bitsandbytes quantizes weights at load time using only the weights themselves, so there is no calibration set. Methods like GPTQ and AWQ do use a small sample of data to choose better rounding, which is one reason they can be more accurate at 4 bits for fixed deployments.

Which GPUs does it support?

It started on NVIDIA GPUs with CUDA, and support for other hardware has been growing. Features and speed differ across backends and library versions, so check the current documentation for your device. Older GPUs may lack some formats or run them slowly.

Can I save a model quantized with bitsandbytes?

Yes. Recent versions of the Transformers and bitsandbytes packages can serialize 8-bit and 4-bit weights so they can be reloaded without re-quantizing. This reduces load time and disk size. Test that the reloaded model gives the same outputs as the one you saved.

Can it help with tabular or table-reading models?

Yes, in the same ways it helps text models. It lowers memory for the backbone and the optimizer when you adapt a large table model, and it fits adapter-based continual learning. Keep input encoders and output heads in higher precision and verify numeric accuracy and probability calibration on held-out tables.

Should I use bitsandbytes for production serving?

It can work, especially for modest traffic, but dedicated serving engines with pre-quantized weights are usually faster and easier to scale. Many teams fine-tune with bitsandbytes and then export to another format for serving. Decide based on measured latency and cost for your workload.

Glossary

• Adapter: A small trainable module placed next to frozen weights.
• Absmax: The largest absolute value in a block, used as its scale.
• bfloat16: A 16-bit format with the range of 32-bit floats and lower precision.
• Block-wise quantization: Quantizing small groups of values with separate scales.
• Code book: The table of allowed values that quantized indices point to.
• Compute dtype: The numeric type used for the actual math after dequantization.
• Dequantization: Expanding compact values back to a higher-precision type.
• Double quantization: Quantizing the scale values themselves.
• FP4: A 4-bit floating-point data type.
• Frozen weights: Weights excluded from gradient updates.
• Gradient checkpointing: Recomputing activations during the backward pass to save memory.
• int8: An 8-bit integer type.
• NF4: A 4-bit data type designed for normally distributed weights.
• Outlier feature: A dimension with unusually large values.
• Paged optimizer: An optimizer whose state can move between GPU and CPU memory.
• Quantization: Representing numbers with fewer bits.
• QLoRA: Fine-tuning adapters on top of a frozen 4-bit quantized model.
• Threshold: The magnitude above which a column is treated as an outlier in LLM.int8.

Summary

bitsandbytes brings 8-bit and 4-bit numbers to PyTorch. It provides quantized linear layers, 8-bit optimizers and paged optimizers, and it powers QLoRA through the NF4 type and double quantization. Its main value is memory: large models fit on small hardware, and fine-tuning becomes affordable.

Use it when you want convenience, no calibration and easy fine-tuning. Use dedicated methods when you need top inference speed on a fixed model. For tabular foundation models, it supports adapter-based adaptation and continual learning on limited hardware, as long as you keep sensitive input and output layers precise and verify numeric quality on held-out tables. Related pages include QLoRA, LoRA, GPTQ, AWQ and TabPFN.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: