Quantisation: 8, 4 and 2 bits, scales, groups and outliers
Quantisation stores weights in fewer bits. The trick is the scale: one per tensor, per row or per small group. Watch the representable values spread out when a few outliers appear.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
The histogram shows the weights of one group with vertical lines at the values the format can represent. The heatmap is the absolute error over the whole 64×64 matrix. The side panels show signal-to-noise, the error of a matrix-vector product and the memory for a whole model.
Things to try
- Raise the outliers with one scale per tensor, then switch to groups of 32.
- Go from 4 bits to 2 bits.
- Compare symmetric and asymmetric (zero-point) groups.
How it works in real systems
GPTQ and AWQ quantise to 4 bits with group size 128 and error-correcting updates; llama.cpp's GGUF formats use small blocks with their own scales; bitsandbytes' NF4 uses levels matched to a normal distribution.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Absmax and zero-point quantisation
import torch
def quant_sym(w, bits=4): # one scale, values in [-2^(b-1), 2^(b-1) - 1]
qmax = 2 ** (bits - 1) - 1
s = w.abs().max() / qmax
return (w / s).round().clamp(-qmax - 1, qmax) * s
def quant_asym(w, bits=4): # scale and zero point, uses the full range
L = 2 ** bits - 1
s = (w.max() - w.min()) / L; z = (-w.min() / s).round()
return ((w / s).round() + z).clamp(0, L).sub(z) * s
w = torch.randn(4096) * 0.02
for f in (quant_sym, quant_asym):
e = w - f(w); print(f.__name__, "SNR dB", (10 * torch.log10(w.pow(2).sum() / e.pow(2).sum())).item())
Step 2 · Groups protect against outliers
def quant_groups(w, bits=4, g=32):
return torch.cat([quant_sym(c, bits) for c in w.split(g)])
w2 = w.clone(); w2[::500] *= 25 # a few outlier weights
for name, q in [("per tensor", quant_sym(w2)), ("groups of 32", quant_groups(w2))]:
e = w2 - q; print(name, (10 * torch.log10(w2.pow(2).sum() / e.pow(2).sum())).item())
Step 3 · Load a model in 4 bits
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch
m = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", device_map="auto",
quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16))
print(m.get_memory_footprint() / 2**30, "GB")
For serving, pre-quantised GPTQ or AWQ checkpoints (vLLM, TGI) or GGUF files (llama.cpp, Ollama) are the usual choices. Always re-run your evaluation set after quantising.