AWQ

Overview

AWQ, short for activation-aware weight quantization, is a post-training method that compresses model weights to 4 bits while keeping accuracy close to the original. Its central insight is simple. Not all weights matter equally. A small share of weight channels, often around one percent, has a large effect on the output, and you can find them by looking at the activations, not at the weights themselves.

Instead of storing those channels in higher precision, which is awkward for hardware, AWQ scales them up before rounding. Scaling up means the rounding error is small relative to the value. The scale is divided out elsewhere, so the model computes the same function. The result is a regular low-bit model that runs fast on common GPU kernels. AWQ sits next to GPTQ, SmoothQuant and TurboQuant in the toolbox of model compression. This article explains how it works, how to use it, and how it applies to models that learn from raw tables.

What it is

AWQ is a weight-only quantization algorithm. Weights are stored with 4 bits (sometimes 3), and activations stay in 16-bit. At run time, a fused kernel reads the packed weights, converts them back to 16-bit values, and multiplies.

Its defining features:

• Activation-aware. Weight importance is judged by the size of the activations that multiply each weight channel.
• Scaling instead of mixed precision. Important channels are protected by a per-channel scale, not by keeping them in a different format.
• No backpropagation. It needs no training and no reconstruction loop. It searches a small number of scale values.
• Small calibration need. It uses a modest calibration set and is less prone to overfit to it than methods that reconstruct layer outputs.
• Hardware friendly. The final model is uniformly low-bit, which keeps kernels simple.

An everyday analogy: you must ship a box of glassware and a box of towels using thin packing foam. Instead of using different foam types, you wrap the glassware in an extra layer of towel first. The packing stays uniform, yet the fragile items get more protection. The scale in AWQ is that extra layer.

What the result looks like

After AWQ, the checkpoint holds packed integer weights, scales and zero points per group, and the per-channel adjustments folded into neighboring layers. In practice, a group size of 128 and 4-bit weights are the common choice.

What it is not

• It is not a training method.
• It is not an activation quantizer. Activations stay in higher precision.
• It is not a hardware format. It produces weights that can be stored in several formats.
• It does not guarantee lossless results. It reduces the loss, particularly for instruction-tuned and chat models.

Where it came from

The need for AWQ came from a pattern seen in large language models. Plain round-to-nearest 4-bit quantization damages quality, and the damage is concentrated. Researchers noticed that keeping a tiny fraction of weights in 16-bit recovers most of the lost quality. The surprising part was which weights to keep: choosing by weight magnitude helped little, while choosing by activation magnitude helped a lot.

That observation led to the AWQ paper in 2023, which later appeared at MLSys in 2024 and received a best paper award there. The paper made two contributions. First, an analysis showing that salient channels are identified by activations. Second, a way to protect them with equivalent scaling, avoiding mixed-precision storage. The authors also released an inference system, called TinyChat, to show that the format runs fast on desktop and edge GPUs.

AWQ builds on earlier work:

• Round-to-nearest quantization, the baseline that every method must beat.
• GPTQ, which uses second-order information to compensate for errors, and its lineage in pruning research.
• LLM.int8() in 2022, which showed the role of outlier dimensions.
• SmoothQuant in 2022, which introduced the idea of an equivalent per-channel scaling between activations and weights. AWQ uses a related transformation but for a different purpose.

Since then, the format has been adopted widely. Libraries such as AutoAWQ and llm-compressor can produce AWQ models, and engines such as vLLM can load them. For the wider research context of this site, see /model/research.

How it works

Step 1: Observe the activations

Run a small calibration set through the model. For each linear layer, record the average magnitude of the input activation on every input channel. Call this vector s_x.

A channel with a large average activation will multiply its weights by big numbers. Any error in those weights is amplified by the same big numbers.

Analogy: in a choir, the loud singers matter most to the overall sound. A small slip by a loud singer is heard. The same slip by a quiet singer is not.

Step 2: Identify the salient channels

The channels with the largest activation magnitudes are the salient ones. In many models they are a small minority, often about one percent. Their weights are the ones whose rounding error hurts the output most.

Step 3: Why not keep them in 16-bit?

Keeping a few columns in 16-bit and the rest in 4-bit is called mixed precision. It does reduce error, but the memory layout becomes irregular, and GPU kernels slow down. A hardware-friendly model should have one format throughout.

Step 4: Scale the salient channels up

Multiply the weight column of each input channel j by a factor s_j, and divide the matching input activation by the same factor. The product is unchanged in exact arithmetic:

• original: output = W times x
• after scaling: output = (W times diag(s)) times (diag(1/s) times x)

Now the weights of salient channels are larger before they are rounded. The rounding step has a fixed size within a group, so a larger value means a smaller relative error. The divided-out activation is not quantized, so it carries no extra error.

Analogy: measuring a tiny object with a coarse ruler gives a poor reading. Magnifying the object first and dividing the reading afterward gives a better one.

Step 5: Choose the scales by a small search

How large should s_j be? Too small and the channel is under-protected. Too large and the other channels in the same group get squeezed, because the group's range grows and their rounding steps become coarse.

AWQ chooses scales from a simple family: s_j equals the activation magnitude s_x_j raised to a power alpha. It then tries alpha values on a grid between 0 and 1, quantizes the layer for each, and picks the alpha with the lowest output error on the calibration data. This is a one-dimensional search per layer. It needs no gradients.

Step 6: Optionally search the clipping range

After scaling, AWQ can also search for a clipping range for each group, so that a rare extreme value does not stretch the grid. This is another small grid search that minimizes output error.

Step 7: Fold the scales away

The division by s_j on the activation side has to happen somewhere. In a transformer, the layer before the linear layer is often a normalization layer or another linear layer. The inverse scale can be folded into its weights. After folding, no extra operation is needed at run time.

Step 8: Quantize and pack

With scales applied, plain round-to-nearest in groups gives the final integer codes. They are packed and stored with per-group scale and zero point.

Step 9: Run with fused kernels

At inference, kernels unpack the 4-bit codes and multiply. On GPUs, the speedup arises mostly because weights are read faster from memory during generation.

Why not reconstruct outputs?

Methods like GPTQ adjust weights to reproduce layer outputs on the calibration data. That can fit the calibration set too closely. AWQ only changes a few scale parameters, so it has far less freedom to overfit. This helps when the real data differs from calibration data, and when the model is a multi-modal or instruction-tuned one.

Key concepts and terms

A worked example

A numeric walk-through

Take a layer with two input channels. The weights for one output are 0.30 for channel A and 0.30 for channel B. Suppose the grid step in the group is 0.10, so rounding error is at most 0.05 on each weight.

The activations are very different. Channel A usually has value 10, and channel B usually has value 0.5.

• Worst-case error from A: 0.05 times 10 = 0.50.
• Worst-case error from B: 0.05 times 0.5 = 0.025.

Channel A is salient. Its errors are twenty times more harmful.

Now scale channel A up by 2 before quantization, which makes its weight 0.60, and divide its activation by 2, giving 5. Suppose the grid step stays at 0.10 because channel B's range still fits within the group. The worst-case error on the scaled weight is still 0.05, but it is now multiplied by an activation of 5.

• New worst-case error from A: 0.05 times 5 = 0.25.

The error from A is halved. If scaling grows too much, the group range grows, the step size rises, and channel B suffers. That trade-off is why AWQ searches for the scale instead of picking a fixed value.

A short Python sketch

This function mimics the per-layer search. It is for learning, not for production.

import torch

def quant_group(w, bits=4):
    maxq = 2 ** bits - 1
    wmin, wmax = w.min(dim=-1, keepdim=True).values, w.max(dim=-1, keepdim=True).values
    scale = (wmax - wmin).clamp(min=1e-8) / maxq
    zero = torch.round(-wmin / scale)
    q = torch.clamp(torch.round(w / scale) + zero, 0, maxq)
    return scale * (q - zero)

def awq_search(W, X, grid=20, bits=4):
    # W: (out, in), X: (n, in)
    s_x = X.abs().mean(dim=0)              # activation magnitude per channel
    ref = X @ W.T                          # original output
    best_err, best_scales = float("inf"), None
    for i in range(grid):
        alpha = i / grid
        s = s_x.pow(alpha).clamp(min=1e-4)
        s = s / (s.max() * s.min()).sqrt() # keep scales balanced
        Wq = quant_group(W * s) / s        # scale, quantize, unscale
        err = (X @ Wq.T - ref).pow(2).mean().item()
        if err < best_err:
            best_err, best_scales = err, s
    return best_scales

In real implementations, the grouping, the folding of scales into neighbors and the clipping search are handled for you.

Using a library

A typical workflow with an open-source library looks like this.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "your-base-model"
quant_path = "model-awq-4bit"
quant_config = {"zero_point": True, "q_group_size": 128,
                "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

Loading the result in an inference engine such as vLLM or through Hugging Face transformers needs no calibration, since the settings are saved with the weights.

Memory arithmetic

A quick calculation helps with planning.

• A 7 billion parameter model needs about 14 GB for 16-bit weights.
• At 4 bits the codes take about 3.5 GB.
• With a group size of 128, scales and zero points add roughly 0.16 bits per weight.
• The real cost is therefore close to 4.2 bits per weight.

A smaller group size improves accuracy and raises this overhead. A group size of 64 doubles the overhead.

Remember that the cache of past keys and values, the activations and the runtime also need memory.

Why it works in plain words

Think of a bridge made of many cables. A few cables carry most of the load. If you must replace all cables with thinner ones, you first reinforce the busy cables, so that the thinner replacement still holds.

In a neural network, the busy cables are the channels fed by large activations. Scaling them up before rounding is the reinforcement step.

The second reason it works is that the model keeps its original structure. No weight is changed in a way that depends on a fragile reconstruction, so the result transfers to new inputs.

Why it can fail

• The calibration data does not excite the channels that matter in production, so the wrong channels are protected.
• The scale search picks a value that is good on average but poor on rare inputs.
• A kernel mismatch causes errors that look like quantization damage.
• The model has layers where the inverse scale cannot be folded for free.

Each of these shows up in testing if you look at the right metrics.

Variants and related ideas

• Different bit widths. The same idea is used for 3-bit and 8-bit weights.
• Different group sizes. Per-channel and small-group variants trade accuracy for memory.
• Clipping search. Searching the range per group often adds a small gain.
• Combination with rotations. Spreading outliers with an orthogonal transform before scaling can help at very low bits.
• Combination with error compensation. Apply scaling first, then an error-compensation step like the one in GPTQ.
• Multi-modal use. Because AWQ overfits less, it has been applied to vision-language models, where calibration data is hard to match.

Checking the result

• Measure perplexity or loss on held-out data.
• Run your real tasks and compare with the 16-bit model.
• Compare memory use and token throughput on target hardware.
• Inspect a sample of outputs for fluency and correctness.
• If the model is multi-modal or tool-using, test those paths separately.

Strengths and limitations

Strengths

• High accuracy at 4 bits. It often matches or beats other weight-only methods on instruction-tuned models.
• Fast to quantize. There is no reconstruction loop, so quantization is quick.
• Small calibration data. A few hundred samples are usually enough.
• Robust to domain shift. Because it changes only scales, it overfits the calibration set less.
• Simple deployment. The final model has one uniform format.
• Memory saving. Weight memory drops to about a quarter of 16-bit.
• Good kernel support. Mature GPU kernels exist for the format.

Limitations

• Weight-only. Activations stay in 16-bit, so compute-bound workloads gain less.
• Depends on activation statistics. If calibration data does not show the real activation pattern, the wrong channels may be protected.
• Folding requires structure. Scales must be folded into a preceding layer. Unusual architectures need custom handling.
• Very low bits remain hard. Below 4 bits, quality drops and further tricks are needed.
• Format fragmentation. Different tools use different packing layouts, so a checkpoint may not load everywhere.
• Gains are workload dependent. At large batch sizes the benefit over 16-bit may shrink.

Quick facts

• Typical setting: 4-bit weights, group size 128, with zero point.
• Typical calibration: a few hundred samples.
• Typical protected share: around one percent of channels.
• Typical search: about twenty alpha values per layer.
• Typical cost: minutes to an hour on one GPU for a mid-size model.
• Typical gain: about four times smaller weights and faster memory-bound decoding.

These are common starting points, not guarantees. Measure on your own model.

Where AWQ is a good choice

• Decoding-heavy serving with small batches.
• Edge or desktop GPUs with limited memory.
• Instruction-tuned and chat models.
• Cases where calibration data is scarce or may not match production.

How it compares with alternatives

AWQ versus GPTQ

Both target 4-bit weights. GPTQ changes the weights themselves to repair errors, and AWQ changes how the weights are scaled before rounding. GPTQ can reach strong results at lower bit widths with enough care. AWQ often generalizes better and is faster to produce. The two can also be seen as complementary, since scaling and error compensation address different causes of error.

AWQ versus SmoothQuant

SmoothQuant also uses per-channel scaling, but its goal is to make activations easy to quantize so that both weights and activations can be 8-bit. AWQ keeps activations in 16-bit and uses scaling to protect important weights at 4-bit. They solve different problems and can be combined in some pipelines.

Choosing in one minute

• Need 4-bit weights fast, with few calibration worries: try AWQ.
• Need to push to 3 bits and can spend time: try GPTQ with small groups.
• Need integer math at large batch: try SmoothQuant.
• Need to shrink the cache: look at TurboQuant.
• Need to run on a laptop CPU: look at GGUF.

AWQ versus formats such as GGUF

GGUF with llama.cpp is a file format and a runtime built for CPUs and mixed hardware, with many quantization types. AWQ is an algorithm that decides how to quantize weights well. The ideas are related, and some tools use activation statistics to improve block formats.

Role in tabular foundation models

Tabular foundation models learn from raw tables rather than from text. A table has rows and columns, mixed types, missing values, and often meaningless or absent column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on many tables, often synthetic ones, and then predict on a new table through in-context learning. The training rows of the new table are given as context, and predictions for the test rows come out of one forward pass. See /model/architecture for how this site describes such models.

AWQ is relevant for a very concrete reason: tabular transformers have the same kind of linear layers, and the same kind of channel imbalance, as language models.

Salient channels in a table model

In a model that embeds cells, some feature channels will carry strong signals. A numeric column with a wide range, a missing-value indicator, or a special marker for the target column can produce large activations. These are natural salient channels. AWQ's rule, protect the channels with large activations, applies directly. The activation statistics should come from real table inputs, which is why calibration data matters.

Calibrate with tables, not text

Use calibration sets made of tables that cover:

• numeric columns with different scales, skew and outliers,
• categorical columns of low and high cardinality,
• several missing-value rates,
• different numbers of rows and columns,
• the tasks you serve: classification, regression, imputation, anomaly detection and semantic labeling.

If you calibrate on the wrong kind of input, AWQ may protect channels that matter for text but not for tables.

Why AWQ's low overfitting helps

Tables vary enormously. Pre-training on synthetic tables gives a model broad priors, and real deployment may involve tables unlike anything in a calibration sample. Because AWQ adjusts only a small number of scale parameters, it is less likely to bend the model toward the calibration tables. This matches the goal of a general tabular model that must work on unseen tables.

In-context learning and what stays precise

Quantizing weights with AWQ does not change the in-context learning mechanism, the use of context rows to infer the pattern in a new table. It does change the numerical precision of attention and feed-forward blocks. Check that the model still uses the context well, for example by measuring accuracy as the number of context rows grows. Keep the following in higher precision unless tests show it is safe:

• numeric feature encoders,
• missing-value handling layers,
• the final prediction heads for regression and probabilities.

Caches and long contexts

AWQ does not compress caches. When the context holds many rows, memory is dominated by attention state. For that part, see TurboQuant, and for related memory tools see FlashAttention and mixed precision.

Continual learning and adapters

For models that keep learning from new tables, treat the AWQ base as frozen and train small LoRA-style adapters on top, as in QLoRA. Methods for avoiding forgetting, such as experience replay and EWC, act on the adapters or on a full-precision copy.

Imputation, anomalies and sector labels

• Imputation. Evaluate error on masked cells before and after quantization. See imputation networks.
• Anomaly detection. Check score rankings and false alarm rates. See anomaly detection.
• Semantic and sector classification. Assigning a meaning or an industry label to a column or table is closer to text understanding. Test rare classes, since they degrade first.

Report results using the evaluation approach on /model/benchmarks.

Practical guidance and best practices

Choose settings

• Start with 4-bit weights and group size 128.
• Use the zero point option unless your kernel requires symmetric grids.
• Try a smaller group size if accuracy is not enough.
• Keep embeddings and the output head in higher precision if tests show sensitivity.

Choose calibration data

• Use data that resembles production inputs.
• Use a few hundred samples of typical length.
• Include rare but important cases.
• For tabular models, use realistic tables of varied shape.
• Keep a separate evaluation set.

Choose a runtime

• Check that your engine has kernels for the AWQ layout you produced.
• Benchmark at your real batch sizes and sequence or row counts.
• Compare with an 8-bit baseline.
• Consider TensorRT-based or vLLM deployments if you need high throughput.

Validate

1. Held-out loss or perplexity.
2. Real task metrics, including calibration and rank metrics for tables.
3. Rare-class and edge-case behavior.
4. Latency, throughput and memory on target hardware.
5. Output comparison with the 16-bit model.

A decision guide

1. If memory capacity is your limit and batch sizes are small, AWQ is a strong first try.
2. If you need more compression than 4 bits gives, consider other methods and expect to validate more.
3. If you serve at large batches, consider activation quantization such as SmoothQuant.
4. If you run on CPUs, consider a block format such as GGUF.
5. If you need to fine-tune, plan adapters on a frozen base.

Reading the tool output

• Many tools print the chosen alpha for each layer. Values near zero mean scaling was not needed. Values near one mean the layer has strong activation imbalance.
• Large changes in output error between layers point to layers worth keeping in higher precision.
• If every layer picks the same alpha, your calibration set may be too small or too uniform.

Serving notes

• Warm up kernels before measuring latency.
• Test the batch sizes, sequence lengths and row counts you really serve.
• Keep the same tokenizer or feature encoder as the original model.
• For tables, keep the same preprocessing, so numeric scaling and missing-value masks match pre-training.
• Track memory over time, since fragmentation can erase the saving.

A short rollout plan

• First: quantize with defaults and compare quality on a held-out set.
• Second: tune group size and calibration data, then benchmark the real runtime.
• Third: shadow traffic, comparing quantized and full-precision outputs side by side.
• After launch: monitor quality and keep the original model for rollback.

Operational habits

• Store the quantization config and library version with the model.
• Keep the original model for rollback and re-quantization.
• Re-run validation after library or kernel upgrades.
• Monitor quality metrics after launch.

Common mistakes

• Using the wrong calibration data. Text for a table model, or a narrow domain for a general model, protects the wrong channels.
• Trusting perplexity alone. Task metrics can move more than perplexity does.
• Mixing up formats. A checkpoint packed for one kernel may not load in another engine.
• Quantizing sensitive layers without testing. Embeddings, heads and encoders often deserve higher precision.
• Over-scaling. Very large scales squeeze the rest of the group and raise error. This is why the search exists.
• Skipping the folding step. If scales are not folded into the preceding layer, the model is no longer equivalent.
• Comparing against a weak baseline. Compare with a tuned 8-bit model, not only with 16-bit.
• Ignoring batch size. Gains at batch size one do not always hold at large batches.
• Calibrating on test data. This inflates results and hides real error.
• Assuming 3-bit behaves like 4-bit. Quality can fall sharply.

Frequently asked questions

What does AWQ stand for?

AWQ stands for activation-aware weight quantization. It is a post-training method that compresses model weights to low precision, usually 4 bits, by using activation statistics to decide which weight channels deserve extra protection before rounding.

Why does AWQ look at activations instead of weights?

A weight's effect on the output is its value multiplied by the activation that feeds it. A small weight attached to a large activation can matter more than a large weight attached to a tiny one. Looking at activations therefore finds the channels where rounding error would hurt most.

Does AWQ keep important weights in 16-bit?

No. That is the key design choice. Mixed precision is hard for hardware, so AWQ scales important channels up before rounding and folds the inverse scale into a neighboring layer. The final model uses a single uniform low-bit format.

Does AWQ need training or backpropagation?

No. It only runs a calibration set forward through the model, measures activations, and searches a small grid of scale values per layer. This is why it is quick and why it overfits the calibration set less than methods that reconstruct layer outputs.

How is AWQ different from GPTQ?

GPTQ modifies the weights column by column, using an inverse Hessian to compensate for rounding errors. AWQ leaves the weight values alone, apart from per-channel scaling, and relies on activation statistics. Which one works better depends on the model, the bit width and the runtime, so testing both is wise.

Does AWQ speed up inference?

Often yes, especially for generation with small batches, because weights are read from memory faster. For compute-bound workloads with large batches the benefit can be small, and methods that also quantize activations, such as SmoothQuant, may help more.

Can AWQ be used for tabular foundation models?

It can be applied to the linear layers of a tabular transformer. Use calibration data made of realistic tables, keep numeric encoders and output heads in higher precision unless tests show otherwise, and verify calibration of probabilities, numeric error and rare-class behavior after quantization.

Can I fine-tune an AWQ model?

You can train adapters on top of a frozen quantized base, as with QLoRA and LoRA. Updating the packed integer weights directly is not the usual workflow. After training, evaluate the combined model again.

How many calibration samples does AWQ need?

A few hundred typical samples are usually enough because the method only measures average activation magnitudes and searches a handful of scales. More data helps only if it adds variety. The samples should look like the data you will serve.

Is AWQ the same as a file format?

No. AWQ is an algorithm that decides how to scale and round weights. The result is stored in a checkpoint layout that depends on the library and kernel. Different tools may use different layouts, so check compatibility with your runtime.

Glossary

• Activation. The values passed from one layer to the next.
• Activation-aware. Using activation statistics to decide how to treat weights.
• Calibration set. A small sample of inputs used to measure statistics.
• Clipping. Restricting the range of the quantization grid to ignore extremes.
• Equivalent transformation. A change that leaves the function unchanged in exact arithmetic.
• Folding. Merging a scale into the weights of an adjacent layer.
• Group size. The number of weights that share a scale and zero point.
• Kernel. A low-level GPU routine, such as a fused dequantize-and-multiply.
• Mixed precision. Using several numeric formats inside one layer or model.
• Perplexity. A language-model quality score; lower is better.
• Salient channel. An input channel whose weights strongly influence outputs.
• Scale. A multiplier or a step size, depending on context.
• Weight-only quantization. Storing weights in low precision while activations stay in 16-bit.
• Zero point. The integer that represents real zero on an asymmetric grid.

Summary

AWQ compresses model weights to 4 bits by finding the input channels with large activations and protecting them with an equivalent per-channel scaling before rounding. It needs no training, only a small calibration set and a short search, and it produces a uniform low-bit model that runs on standard kernels.

It works best as a weight-only method for memory-bound serving, generalizes well across domains, and complements GPTQ, SmoothQuant and TurboQuant. For tabular foundation models, apply it to the large linear layers, calibrate on realistic tables, keep numeric encoders and output heads precise, and validate calibration, numeric error and rare classes before deployment. See /model/benchmarks for evaluation guidance.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: