SmoothQuant

Overview

SmoothQuant is a post-training quantization method that lets a model run with 8-bit weights and 8-bit activations, often written W8A8. Its key trick is a mathematically equivalent rescaling. Activations in large transformers contain a few channels with huge values, called outliers, which make 8-bit activation quantization hard. SmoothQuant divides those channels by a scale and multiplies the matching weights by the same scale. The layer computes the same function, but the activations become smooth and the weights absorb the difficulty.

Because both sides are 8-bit, the matrix multiplies can use integer arithmetic on the GPU, which raises throughput and cuts memory. This makes SmoothQuant a natural fit for serving at larger batch sizes, where compute rather than memory is the limit. It complements weight-only methods such as GPTQ and AWQ, and it is related in spirit to cache and vector methods such as TurboQuant. This article explains how it works, how to use it, and how it applies to models that learn from raw tables.

What it is

SmoothQuant is a training-free technique that prepares a model for W8A8 quantization. It has two parts:

1. A smoothing step that migrates quantization difficulty from activations to weights, by per-channel scaling.
2. A standard 8-bit quantization of the smoothed weights and activations.

The first part is the new idea. The second uses common integer quantization with scales per tensor or per token and per channel.

Why activations are harder than weights

Weights are static. Their distribution is known in advance, and they tend to be fairly uniform. Activations change with every input, and in large language models a few channels carry values that are tens or hundreds of times larger than the rest. These outliers appear in the same channels across many tokens.

If you use one 8-bit scale for a tensor, the scale must cover the outliers. The normal channels then use only a few of the 256 levels, and their information is lost.

Analogy: imagine measuring both a skyscraper and a houseplant with the same ruler marked in a fixed number of ticks. To fit the skyscraper, the ticks must be far apart, and the houseplant is read as zero. Smoothing is like putting the skyscraper on a smaller scale model before measuring.

What it is not

• It is not a training method.
• It is not a replacement for good kernels. The speedup requires integer matrix-multiply support.
• It is not limited to one model family, but each architecture needs the scales folded in the right places.
• It does not quantize the cache by itself. That is a separate step.

Where it came from

Early work on transformer quantization found that activations, not weights, were the obstacle at 8 bits. LLM.int8() in 2022 showed that a handful of outlier feature dimensions cause the failure of naive 8-bit quantization at large scale. Its remedy was a mixed-precision decomposition: handle outlier dimensions in 16-bit and the rest in 8-bit. That works but is slower and less hardware friendly.

The SmoothQuant paper appeared in 2022 and was presented at ICML in 2023. Its contribution was to avoid mixed precision by rescaling. Since the outliers sit in particular channels, the idea was to flatten them by a per-channel factor that is cancelled in the weights. The authors showed that large models could run W8A8 with little loss, and that this gave real speedups and memory savings with integer kernels.

Later work extended the family:

• AWQ used a related equivalent scaling, but to protect weights at 4 bits.
• GPTQ improved weight-only compression with second-order information.
• Rotation-based methods spread outliers using orthogonal transforms.
• Floating-point 8-bit formats on newer GPUs reduced the pain of outliers, since they have a wider dynamic range.
• Inference stacks such as TensorRT and vLLM added support for SmoothQuant-style models.

For the research context of this site, see /model/research.

How it works

Step 1: Look at the matrix product

Consider one linear layer: output Y equals X times W, where X holds activations with shape tokens by input channels and W holds weights with shape input channels by output channels.

If channel j of X has large values, the product still works, but quantizing X is hard. Channel j of W is the matching row.

Step 2: Insert an equivalent scaling

For any positive per-channel factor s_j:

• Y = X times W
• Y = (X times diag(1/s)) times (diag(s) times W)

The two lines give the same result in exact arithmetic. The first factor has smoothed activations. The second has scaled weights.

Analogy: two friends split a bill. If one pays a large share this time, the other can pay a large share next time, and the total remains the same. Here the "bill" is the quantization difficulty, moved from activations to weights.

Step 3: Choose the scale

SmoothQuant sets the factor from the largest absolute values per channel in activations and weights:

• s_j = max|X_j| raised to alpha, divided by max|W_j| raised to (1 minus alpha)

The migration strength alpha controls the split. With alpha near 0.5, the maximum of each channel is balanced between activations and weights. A larger alpha moves more difficulty to the weights, which helps when activations are very spiky, such as in some very large models.

Step 4: Estimate activation statistics with calibration

To compute max|X_j|, run a small calibration set through the model and record the per-channel maxima. A few hundred samples are common. Because the scales are computed offline, no extra work happens at inference.

Step 5: Fold the scales

The division by s_j must be applied to the layer input. In a transformer, the input to the attention projections and the feed-forward layer follows a normalization layer. The inverse scale can be folded into the normalization layer's weights, so no extra operation is needed at run time. The multiplication by s_j is folded into the linear layer's weights.

Some places, for example the input to the output projection of attention or the second feed-forward layer, do not follow a normalization layer. In such places, implementations may either skip smoothing or add a lightweight scaling operation.

Step 6: Quantize to 8 bits

After smoothing, quantize weights and activations to INT8. Several granularities exist:

• Per-tensor. One scale for the whole tensor. Simple and fastest.
• Per-token for activations, per-channel for weights. More accurate, with a small cost.
• Static or dynamic. Static activation scales are computed once from calibration. Dynamic scales are computed at run time for each input.

Step 7: Run integer matrix multiplies

The GPU multiplies INT8 by INT8, accumulates in INT32, and converts the result back to higher precision using the stored scales. Integer tensor cores are faster than 16-bit ones, and 8-bit data halves the memory traffic.

Efficiency levels

The authors described several levels of quantization. The more aggressive ones use per-tensor, static scales and quantize more of the operations, including batched matrix multiplications inside attention. They are faster but riskier. The gentler ones use finer scales and are more accurate. You choose one by testing.

Why smoothing helps

After smoothing, the outlier channels in X are brought closer to the others. The weights get larger in those channels, but weights are well behaved and easier to quantize per output channel. The sum of the two difficulties is smaller than the activation problem alone.

Key concepts and terms

A worked example

A numeric walk-through

Take one linear layer with two input channels. For one token the activations are x = (40, 1). The weights for one output are w = (0.5, 0.5). The exact output is 40 times 0.5 plus 1 times 0.5 = 20.5.

Suppose we use one INT8 scale for the activation vector, with range up to 127. The largest value is 40, so the scale is 40 divided by 127, about 0.315. Quantizing 1 gives round(1 / 0.315) = round(3.17) = 3, which dequantizes to 0.945. Not bad here, but with a larger outlier, say 400, the scale is about 3.15, and 1 rounds to 0. The small channel vanishes.

Now apply smoothing with s = (10, 1) for the two channels.

• Smoothed activations: (4, 1).
• Scaled weights: (5.0, 0.5).
• Output: 4 times 5.0 plus 1 times 0.5 = 20.5. The same result.

The activation range is now 4, the scale is 4 divided by 127, about 0.031, and the second channel is represented as round(1 / 0.031) = 32. It is kept accurately. The weights now have a larger first entry, 5.0 against 0.5, which is fine because weights can use per-output-channel scales, and the large entry rarely dominates in the same way.

A short Python sketch

This function computes smoothing scales and applies them to a layer. It is a learning aid.

import torch

@torch.no_grad()
def smooth_layer(ln_weight, ln_bias, linear_weight, act_max, alpha=0.5):
    # act_max: (in_features,) max |activation| per input channel
    # linear_weight: (out_features, in_features)
    w_max = linear_weight.abs().max(dim=0).values.clamp(min=1e-5)
    s = (act_max.clamp(min=1e-5) ** alpha) / (w_max ** (1 - alpha))
    s = s.clamp(min=1e-5)

    # divide the normalization output by s
    ln_weight.div_(s)
    if ln_bias is not None:
        ln_bias.div_(s)
    # multiply the matching weight columns by s
    linear_weight.mul_(s.view(1, -1))
    return s

def quantize_int8(t, per_channel_dim=None):
    if per_channel_dim is None:
        scale = t.abs().max() / 127
    else:
        scale = t.abs().amax(dim=per_channel_dim, keepdim=True) / 127
    q = torch.clamp(torch.round(t / scale), -128, 127).to(torch.int8)
    return q, scale

In practice, a library handles all layers, the folding sites and the kernel selection.

Calibration in practice

import torch

act_max = {}

def make_hook(name):
    def hook(module, inputs):
        x = inputs[0].detach().abs().reshape(-1, inputs[0].shape[-1])
        m = x.max(dim=0).values
        act_max[name] = torch.maximum(act_max.get(name, m), m)
    return hook

for name, mod in model.named_modules():
    if isinstance(mod, torch.nn.Linear):
        mod.register_forward_pre_hook(make_hook(name))

with torch.no_grad():
    for batch in calibration_batches:
        model(**batch)

The collected maxima feed the scale formula. Libraries such as llm-compressor and the reference implementation automate these steps and work with PyTorch models loaded through Hugging Face.

Memory and speed arithmetic

• 16-bit weights take 2 bytes each. INT8 weights take 1 byte, so weight memory halves.
• Activations also halve, which matters for long inputs and large batches.
• Integer tensor cores can run about twice as many operations per second as 16-bit ones on many GPUs, so an ideal speedup in the matrix multiplies approaches two times.
• Real speedups are smaller, because normalization, attention softmax, quantize and dequantize steps remain in higher precision.

Always measure end to end. The theoretical ratio is an upper bound.

Why it works in plain words

Imagine a kitchen where one cook uses enormous pans and the others use small ones. A single set of measuring cups cannot serve both well. SmoothQuant gives the big-pan cook a smaller recipe and compensates by scaling up the ingredients elsewhere, so every cook can use the same measuring cups.

In the network, the activation side has the big pans. The weights are calm and can handle a larger range once the scale is moved to them.

Why it can fail

• Calibration did not see the extreme inputs, so static ranges clip them.
• The best alpha differs by layer, and a single global value is a compromise.
• A layer has no preceding normalization, so its scale cannot be folded for free.
• The engine falls back to slow kernels for some operations.

Variants and related ideas

• Per-layer alpha. Choose migration strength layer by layer using calibration error.
• Learned scales. Optimize scales on calibration data with a few gradient steps.
• Rotations. Spread outliers with orthogonal transforms instead of per-channel scales.
• Weight-activation at lower bits. Try 4-bit weights with 8-bit activations, which combines ideas from AWQ and GPTQ.
• Cache quantization. Apply a separate method to keys and values, such as TurboQuant.
• Floating-point 8-bit. Use FP8 formats where hardware supports them, sometimes with a milder smoothing step.

Checking the result

• Compare perplexity or task loss with the 16-bit model.
• Compare throughput at the batch sizes you serve.
• Compare peak memory.
• Test long inputs, since outliers can grow with length.
• Test the exact inference engine you will use in production.

Strengths and limitations

Strengths

• Hardware-friendly. Standard INT8 kernels do the work, with no mixed-precision branches.
• Training-free. A small calibration set is enough.
• Real speedup for compute-bound serving. Integer math and smaller tensors raise throughput at larger batch sizes.
• Memory saving. Weights and activations take half the space of 16-bit.
• Simple and general. The same idea works across many transformer models.
• Composable. It can be combined with cache quantization and with other techniques.

Limitations

• Only 8 bits in the basic form. Going to 4 bits for weights and activations is much harder and needs extra methods.
• Folding sites. Not every linear layer follows a normalization layer, so some layers cannot be smoothed for free.
• Calibration sensitivity. Maxima come from the calibration set. A rare spike in production may exceed the range.
• Per-tensor static scales are fragile. They are the fastest option and also the least accurate.
• Kernel support. You need an engine with INT8 kernels for your GPU.
• Smaller gain at small batches. When decoding is memory-bound, weight-only 4-bit methods may beat W8A8.
• Newer formats. On hardware with native 8-bit floating point, a simpler FP8 approach may be competitive.

Quick facts

• Typical setting: alpha near 0.5, INT8 weights per output channel, INT8 activations per token or per tensor.
• Typical calibration: a few hundred samples.
• Typical cost: minutes on one GPU, since only maxima are collected.
• Typical gain: half the memory for weights and activations, and faster integer matrix multiplies.
• Typical risk: clipping of rare spikes when scales are static.

These are common starting points, not guarantees. Measure on your own model.

When SmoothQuant is a good choice

• High-throughput serving with larger batches.
• Hardware with strong INT8 support.
• Cases where you want one uniform 8-bit pipeline with predictable accuracy.
• Prefill-heavy workloads, where the model processes long inputs at once.

How it compares with alternatives

SmoothQuant versus weight-only methods

Weight-only methods shine when each token requires reading all weights from memory, as in small-batch decoding. SmoothQuant shines when the arithmetic itself is the limit, as in prefill and large-batch decoding. Many deployments need both: smaller weights for memory and integer math for speed. In practice you pick based on measured throughput at your batch sizes.

Choosing in one minute

• Need integer math at large batch: try SmoothQuant.
• Need small weights for memory-bound decoding: try AWQ or GPTQ.
• Need to shrink the cache: look at TurboQuant.
• Need to run on a laptop CPU: look at GGUF with llama.cpp.
• Have FP8 hardware: test FP8 first, then decide whether smoothing adds value.

SmoothQuant versus LLM.int8()

LLM.int8() keeps the outlier dimensions in 16-bit, so it is accurate but needs a more complex kernel. SmoothQuant removes the need for that split. It is simpler to deploy, and the kernel is pure INT8.

SmoothQuant versus AWQ

Both use equivalent per-channel scaling. SmoothQuant scales to make activations quantizable. AWQ scales to protect important weights at low bit width. The target and the formula differ.

Role in tabular foundation models

Tabular foundation models learn from raw tables instead of text. A table has rows and columns, mixed types, missing values, and column names that may be missing or meaningless. Models such as TabPFN, TabICL and TabSTAR are pre-trained on many tables, often synthetic ones, and then use in-context learning: the training rows of a new table are context, and predictions for test rows come out of one forward pass. See /model/architecture for how this site describes such models.

SmoothQuant is especially relevant because tabular inputs can produce activation outliers even more readily than text.

Why tables create activation outliers

Raw tables hold values of very different scales: an identifier in the millions next to a rate between 0 and 1. Even after normalization, heavy-tailed columns, missing-value indicators and special markers can produce a few feature channels with very large activations. Those are the same kind of outlier channels that SmoothQuant was designed for. A table model may therefore benefit from smoothing, but only a measurement can say how strong the effect is.

Row and column attention is compute-heavy

Tabular transformers often alternate attention across the rows and across the columns. Attention over many rows is expensive, and serving many tables at once is compute-bound. That is exactly the regime in which W8A8 integer arithmetic helps. It reduces compute time per table and lets one GPU serve more requests.

Calibration must reflect real tables

The scale statistics come from calibration inputs. For tabular models, those inputs should be tables, not text, and should cover:

• numeric columns with different ranges and heavy tails,
• categorical columns of various cardinalities,
• several missing-value rates,
• different row and column counts,
• the tasks you serve.

Because SmoothQuant records maxima, a rare extreme value in production can exceed the calibrated range and be clipped. Include extreme tables in calibration, or use dynamic per-token scales for safety.

Preprocessing still matters

Quantization sits after the preprocessing that turns raw cells into model inputs. Clean, consistent numeric scaling and a clear missing-value representation make activations better behaved, and can reduce the need for aggressive smoothing. Quantization is not a substitute for good preprocessing.

Numeric output needs care

Regression, probability estimates and imputed values are numbers that people use directly. After W8A8 conversion, verify numeric error, calibration of probabilities and rank metrics, not only accuracy. Keep the final prediction heads and numeric encoders in higher precision unless tests show they are safe.

In-context learning and caches

Quantizing the weights and activations does not change how in-context learning works, but it changes the numeric precision of each pass. Test accuracy as the number of context rows grows. For the state held across many rows, see TurboQuant and FlashAttention. For related ideas on number formats, see mixed precision.

Continual learning and adapters

If a tabular model keeps learning from new tables, use small LoRA adapters on top of a frozen quantized base, as in QLoRA. Techniques such as experience replay and EWC work on the adapters or on a full-precision copy. If the base is re-quantized, recompute the smoothing scales.

Imputation, anomalies and sector labels

• Imputation. Check the error on masked cells. See imputation networks.
• Anomaly detection. Check score rankings and false-alarm rates. See anomaly detection.
• Semantic and sector classification. Check rare classes first, since they degrade first.

Report results using the approach described at /model/benchmarks.

Practical guidance and best practices

Choose the setting

• Start with alpha 0.5 and per-token dynamic activation scales with per-channel weight scales.
• If accuracy is fine, try static per-tensor scales for more speed.
• If activations are extremely spiky, increase alpha toward 0.8 and test again.
• Keep sensitive layers in higher precision if needed.

Choose calibration data

• Use inputs from the real distribution, including extremes.
• Use a few hundred samples of typical size.
• For tabular models, use realistic tables of varied shape.
• Keep a separate evaluation set.

Choose the runtime

• Check INT8 kernel support on your GPU and engine, such as TensorRT or vLLM.
• Benchmark at real batch sizes and context lengths.
• Compare with a weight-only 4-bit model and with FP8 if available.
• Consider ONNX Runtime if you deploy to varied hardware.

Validate

1. Perplexity or task loss on held-out data.
2. Real task metrics and, for tables, calibration and numeric error.
3. Behavior on extreme inputs.
4. Throughput, latency and memory.
5. Output comparison with the 16-bit model.

A decision guide

1. Is your serving compute-bound? Try SmoothQuant W8A8.
2. Is it memory-bound with small batches? Try a weight-only 4-bit method first.
3. Do you have FP8 hardware? Test FP8 as a simpler option.
4. Do you need to fine-tune? Use adapters on a frozen base.
5. Are inputs wildly variable? Prefer dynamic scales.

Reading the diagnostics

• Plot the per-channel maxima of each layer before and after smoothing. A good result shows a flatter profile.
• Look at the share of activations that hit the clip limit. A few in a million is fine. Many is a warning.
• Compare layer-wise output error across alpha values. A sharp minimum suggests a good choice. A flat curve means the setting matters little.

Serving notes

• Warm up kernels before measuring.
• Measure at the real batch sizes and context lengths.
• Check that the engine uses fused INT8 kernels, not a fallback.
• Keep preprocessing identical to the original model.
• For tables, keep numeric scaling and missing-value masks identical to pre-training.

A short rollout plan

• First: calibrate with defaults and compare quality on a held-out set.
• Second: test dynamic and static scales and measure throughput.
• Third: shadow traffic against the full-precision model.
• After launch: watch for clipping rates and quality drift.

Operational habits

• Store alpha, the calibration description and library versions with the model.
• Keep the original model for rollback.
• Re-run validation after kernel or library upgrades.
• Monitor output quality and range overflow in production.

Common mistakes

• Calibrating on the wrong inputs. Text for a table model, or only typical inputs without extremes, leads to clipped activations later.
• Using static per-tensor scales without testing. They are fast, but can be the least accurate.
• Ignoring where scales can be folded. Layers without a preceding normalization may need a separate scaling step.
• Picking alpha blindly. The best value depends on how spiky the activations are.
• Expecting speedups without kernels. Without INT8 support the model may run slower than before.
• Forgetting the attention matmuls. Aggressive levels also quantize operations inside attention, which needs testing.
• Judging only by perplexity. Task metrics and rare cases matter.
• Applying smoothing twice. Re-smoothing an already smoothed model changes the function if scales are folded incorrectly.
• Skipping preprocessing fixes. Poor numeric scaling of raw columns makes outliers worse.
• Not testing long inputs or many rows. Outliers can grow with length.

Frequently asked questions

What is SmoothQuant in simple terms?

SmoothQuant is a way to run a neural network with 8-bit weights and 8-bit activations. It first rescales each input channel so that the large activation values shrink and the matching weights grow by the same factor. The layer output does not change, but both sides become easy to round to 8 bits.

What problem does SmoothQuant solve?

It solves the outlier problem. A few activation channels in large transformers have values far larger than the rest, so a single 8-bit scale wastes most of its levels and loses information. By moving the range from activations to weights, SmoothQuant makes 8-bit activation quantization practical.

Does SmoothQuant require training?

No. It is a post-training method. It needs a small calibration set to measure the per-channel maximum of activations, then it computes scales with a simple formula and folds them into the model. No gradient updates are involved.

What does the alpha parameter do?

Alpha controls how much of the quantization difficulty moves from activations to weights. A value around 0.5 balances the two. A higher value moves more difficulty into the weights and helps when activations are very spiky. The best value is found by testing on your own model and data.

Is SmoothQuant better than GPTQ or AWQ?

They solve different problems. GPTQ and AWQ compress weights to 3 or 4 bits and help when memory bandwidth is the limit. SmoothQuant enables 8-bit integer math for both weights and activations and helps when compute is the limit. Many teams choose according to their batch sizes and hardware.

Does SmoothQuant make inference faster?

It can, if the runtime has INT8 kernels for your GPU. Integer matrix multiplies are faster than 16-bit ones, and 8-bit tensors cut memory traffic. The gains are largest for compute-bound work such as large batches and long prompts, and smaller for small-batch decoding.

Can SmoothQuant be used for tabular foundation models?

Yes, in principle. Tabular transformers have linear layers and can show outlier channels, especially with heavy-tailed numeric columns. Calibrate with realistic tables, include extreme values, keep numeric encoders and output heads precise, and check calibration, numeric error and rare-class behavior afterward.

Can SmoothQuant be combined with other methods?

Yes. It can be combined with cache quantization such as TurboQuant, with adapters for fine-tuning such as LoRA, and with attention optimizations such as FlashAttention. Each combination should be tested, since errors from different steps can add up.

What happens if production data has bigger values than the calibration data?

With static scales, values beyond the calibrated range are clipped, which can hurt accuracy. To reduce the risk, include extreme cases in calibration, use dynamic per-token scales, or add range checks and fall back to higher precision when a spike is detected.

Is SmoothQuant still useful with 8-bit floating-point hardware?

Floating-point 8-bit formats have a wider dynamic range, which makes outliers less damaging, and they can be simpler to use. SmoothQuant remains useful where integer kernels are the best-supported option or where its scaling still improves accuracy. Measure both on your hardware.

Glossary

• Activation. The values passed from one layer to the next.
• Alpha. The migration strength that sets how much difficulty moves to weights.
• Calibration set. A small sample used to measure activation ranges.
• Dynamic quantization. Computing scales at run time for each input.
• Equivalent transformation. A change that leaves the function unchanged.
• Folding. Merging a scale into the weights of a neighboring layer.
• GEMM. General matrix multiply, the core operation of linear layers.
• INT8. An 8-bit integer number format.
• Outlier. A value far larger than typical values in the same tensor.
• Per-channel scale. A separate scale for each channel.
• Per-tensor scale. One scale for the whole tensor.
• Per-token scale. A separate scale for each token.
• Smoothing. The rescaling step that moves difficulty from activations to weights.
• Static quantization. Using scales fixed ahead of time from calibration.
• W8A8. 8-bit weights and 8-bit activations.

Summary

SmoothQuant makes 8-bit weights and activations practical by rescaling channels so that activation outliers shrink and weights absorb the range, without changing what the layer computes. It needs only a small calibration set, runs on standard INT8 kernels, and gives real gains for compute-bound serving.

It complements weight-only methods such as GPTQ and AWQ, and cache methods such as TurboQuant. For tabular foundation models, calibrate on realistic tables including extremes, keep numeric encoders and output heads precise, and validate numeric error, calibration and rare classes. See /model/benchmarks for evaluation guidance and /model/architecture for the model context.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: