GPTQ

Overview

GPTQ is a post-training quantization method that compresses the weights of a trained neural network to 4 bits, 3 bits or even 2 bits per weight. It does this one layer at a time. It uses approximate second-order information, a Hessian built from a small set of example inputs, to decide how to round each weight so that the layer output barely changes. No retraining is needed.

The practical result is a model that is about four times smaller than its 16-bit original and often close to it in quality. That means cheaper GPUs, lower memory use and faster generation when memory bandwidth is the bottleneck. GPTQ became one of the standard ways to shrink large transformers, and it sits in the same family as AWQ, SmoothQuant and TurboQuant. This article explains the method, the vocabulary, a worked example, and how it applies to models that learn from raw tables.

What it is

GPTQ stands for a quantization algorithm designed for generative pre-trained transformers. The name reflects its first target, but the method works on any stack of linear layers. It is a weight-only method in its common form. The weights are stored in low precision and are expanded back to 16-bit numbers on the fly during the matrix multiply. The activations, the numbers that flow between layers, stay in 16-bit.

Three ideas define it.

• It is post-training. You start from a finished model and a small calibration set. You do not run gradient descent on the full model.
• It is layer-wise. Each linear layer is treated as its own small problem: find low-bit weights that reproduce the original layer output on the calibration inputs.
• It is error-aware. When one weight is rounded, the rounding error is pushed onto the weights that have not been rounded yet, so the layer output stays close to the original.

An everyday analogy helps. Imagine you must round every price on a long restaurant bill to the nearest whole unit. If you round each line alone, the errors may add up to a noticeable gap at the bottom. A careful accountant rounds the first line, sees the gap, and nudges the next lines slightly so the total stays right. GPTQ is that careful accountant, working on the sums inside a neural network layer.

What the output looks like

After quantization you get a file with packed integer weights, plus a scale and often a zero point for each small group of weights. A typical setting is 4-bit weights with one scale per group of 128 weights. Several 4-bit values are packed into one 32-bit integer. A special GPU kernel reads the packed data and dequantizes it inside the matrix multiply.

What it is not

• It is not a training method. There is no loss over a dataset and no optimizer.
• It is not activation quantization. Activations remain in higher precision unless another technique is added.
• It is not pruning, although it grew out of pruning research.
• It is not a file format. Several formats and runtimes can store GPTQ-style weights.

Where it came from

The roots go back to work on pruning. In the early 1990s, a method called Optimal Brain Surgeon used the inverse of the Hessian of the loss to decide which weights to remove and how to adjust the remaining weights to compensate. An earlier method, Optimal Brain Damage, used a diagonal approximation of the same idea. Both showed that curvature information beats naive magnitude-based choices.

Decades later, researchers adapted these ideas to the layer-wise setting. Work on Optimal Brain Compression and Optimal Brain Quantization in 2022 showed that you could solve a layer-wise reconstruction problem greedily and update the remaining weights after each decision. That approach was accurate but slow, and it did not scale to models with billions of parameters.

The GPTQ paper appeared in 2022 and was presented at ICLR in 2023. Its contribution was to make the idea fast enough for very large models. It did this with three changes: a fixed column order instead of a greedy order, lazy batched updates, and a Cholesky-based way to handle the inverse Hessian. With these, models with tens or hundreds of billions of parameters could be quantized in hours on a single GPU.

Around the same time, other work attacked the same problem from different angles. LLM.int8() in 2022 showed that a few outlier dimensions break naive 8-bit quantization and proposed mixed-precision decomposition. SmoothQuant in 2022 moved activation outliers into the weights. AWQ in 2023 protected the weights that matter most for the activations. The open-source community built tools such as bitsandbytes, llama.cpp with the GGUF format, and ExLlamaV2 around these ideas. For the wider research context of this site, see /model/research.

How it works

Step 1: Pick a layer and collect calibration inputs

Take one linear layer with weight matrix W. Run a small calibration set through the model up to that layer and record the inputs X. A common choice is a few hundred text sequences of 2,048 tokens each. The number is not magic. It needs to be large enough that the statistics of X are stable.

Analogy: before tailoring a suit, the tailor needs a few measurements of the person who will wear it. The calibration set is that measurement session.

Step 2: State the goal

For each layer GPTQ solves this problem: find low-bit weights that make the layer output on X as close as possible to the original output. In symbols, minimize the squared difference between W X and Q X, where Q holds only values from the low-bit grid.

This is a different goal from "make each weight close to its original value". Two weights with the same rounding error can matter very differently, depending on how much the inputs excite them. The goal is about outputs, not weights.

Step 3: Build the Hessian

For this squared error, the Hessian with respect to one row of W is proportional to X multiplied by its transpose. It is a square matrix with one row and column per input feature. It is the same for every row of W, so you compute it once per layer.

Think of the Hessian as a map of which input features move together and how strongly. If two input features are almost identical, the layer cannot tell them apart, and the error can be shifted from one to the other at little cost.

Step 4: Add damping

The Hessian can be nearly singular, for example when some features are always zero or perfectly correlated. GPTQ adds a small value, typically one percent of the average diagonal, to the diagonal. This keeps the inverse stable. It is like putting a small rubber pad under a wobbling table.

Step 5: Quantize column by column

Now the main loop. Go through the columns of W in a fixed order. For each column:

1. Quantize the column to the nearest value on the low-bit grid.
2. Compute the error between the original and quantized column.
3. Divide that error by the matching diagonal entry of the inverse Hessian.
4. Subtract the scaled error from all the columns that have not yet been quantized, using the matching row of the inverse Hessian.

Step 4 is the heart of the method. The remaining weights absorb the damage and compensate for it.

Step 6: Use a Cholesky factor

Updating the inverse Hessian after every column is expensive. GPTQ instead computes the Cholesky decomposition of the inverse Hessian once. The rows of this factor contain exactly the numbers needed for each step, so no repeated inversion is needed. This is both faster and more numerically stable.

Step 7: Process in blocks

Updating all the remaining columns after every single column wastes memory bandwidth. GPTQ therefore works on blocks of columns, often 128 at a time. Inside a block the updates are applied immediately. The updates to columns outside the block are collected and applied once when the block ends. This "lazy batch" trick makes the algorithm limited by compute rather than by memory traffic.

Step 8: Save scales and pack

For every group of weights, a scale and a zero point are stored. The integer codes are packed, for example eight 4-bit codes per 32-bit word. At inference, a fused kernel unpacks, scales and multiplies.

Step 9: Repeat for every layer

GPTQ moves through the transformer block by block. A common option is "true sequential" ordering: quantize the attention projections first, then recompute the inputs for the following layers using the already quantized ones. This lets later layers see the real quantized inputs, which reduces accumulated error.

Optional refinements

• Group size. A smaller group gives more scales, which improves accuracy but uses more memory. Groups of 128 are common; 64 or 32 appear for very low bit widths.
• Act-order. Process columns in descending order of activation magnitude (the diagonal of the Hessian). Important columns are quantized first, while many unquantized weights are still free to compensate. This can improve accuracy a lot, but it makes the memory layout less regular and can slow some kernels.
• Symmetric or asymmetric grids. Symmetric grids are simpler; asymmetric grids use a zero point and fit skewed ranges better.
• Outlier handling. Some implementations keep a few sensitive layers, such as the output head, in higher precision.

Key concepts and terms

A worked example

A tiny numeric walk-through

Suppose a layer has one output and two input features. The weights are 0.40 and 0.31. Suppose our grid only has multiples of 0.25, so the nearest values are 0.50 and 0.25.

Plain rounding gives 0.50 and 0.25. The errors are +0.10 and -0.06.

Now suppose the two features are strongly correlated, so that the Hessian says they are nearly interchangeable. After rounding the first weight to 0.50, the error is +0.10 on the first weight. GPTQ pushes a correction onto the second weight: because feature one is now too large, the second weight is reduced. If the correlation is near one, the second weight might become about 0.21, which rounds to 0.25 with a small remaining error, but the combined output is much closer to the original than plain rounding produced.

The key point is that the size of the correction depends on the inverse Hessian. Correlated features share the blame, and independent features do not.

A minimal Python sketch

This short function shows the core loop. It is educational and not tuned for speed.

import torch

def quantize_to_grid(w, scale, zero, maxq):
    q = torch.clamp(torch.round(w / scale) + zero, 0, maxq)
    return scale * (q - zero)

def gptq_layer(W, X, bits=4, damp=0.01):
    # W: (out_features, in_features), X: (n_samples, in_features)
    W = W.clone().float()
    H = 2.0 * X.T @ X / X.shape[0]
    diag = torch.arange(H.shape[0])
    H[diag, diag] += damp * H.diag().mean()

    # upper Cholesky factor of the inverse Hessian
    Hinv = torch.linalg.cholesky(torch.cholesky_inverse(
        torch.linalg.cholesky(H)), upper=True)

    maxq = 2 ** bits - 1
    wmin = W.min(dim=1, keepdim=True).values
    wmax = W.max(dim=1, keepdim=True).values
    scale = (wmax - wmin).clamp(min=1e-8) / maxq
    zero = torch.round(-wmin / scale)

    Q = torch.zeros_like(W)
    for i in range(W.shape[1]):
        w = W[:, i:i + 1]
        q = quantize_to_grid(w, scale, zero, maxq)
        Q[:, i:i + 1] = q
        err = (w - q) / Hinv[i, i]
        W[:, i + 1:] -= err @ Hinv[i:i + 1, i + 1:]
    return Q

Compare the output error of this function against plain rounding on random data with correlated features, and you will see the gap grow as the bit width falls.

Using a library

Most people run GPTQ through a library rather than writing it. With the Hugging Face transformers integration, quantizing a model looks like this.

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_id = "your-base-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)

gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer,
                         group_size=128)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=gptq_config,
    device_map="auto",
)
model.save_pretrained("model-gptq-4bit")

Loading the saved model later needs no calibration step. The quantization config is stored with the weights, and inference engines such as vLLM can load many GPTQ checkpoints directly.

Memory arithmetic

A quick calculation helps you plan hardware.

• A model with 7 billion parameters in 16-bit needs about 14 GB for weights.
• The same model at 4 bits needs about 3.5 GB for the codes.
• Scales and zero points add a little. With a group size of 128, a 16-bit scale and a 4-bit zero point per group add roughly 0.16 bits per weight.
• So the real cost is close to 4.16 bits per weight, not exactly 4.

Smaller groups raise this overhead. With a group size of 32 the overhead is about four times larger.

Activations, the cache of past keys and values, and the runtime itself need memory on top of the weights.

What the error looks like in practice

Quantization error is not spread evenly.

• Layers near the start and end of the network are often more sensitive.
• Layers with large outlier channels suffer more.
• Weights attached to rarely used features matter little, and the Hessian knows it.
• Output quality tends to fall gradually, then suddenly, as bits go down.

This is why a bit width that looks fine on average can fail on a specific task. Always test the cases you care about.

Checking the result

A quantized model must be tested, not trusted. A reasonable check list:

• Compare perplexity on a held-out text set against the 16-bit model.
• Run the downstream tasks you actually care about.
• Compare latency and memory on your real hardware.
• Look at a sample of outputs by hand, especially long ones.

Strengths and limitations

Strengths

• Strong accuracy at 4 bits. For many large models the quality loss is small, and larger models tend to tolerate quantization better than small ones.
• No training. It needs a few hundred examples and a single GPU, not a full training cluster.
• Fast to run. Quantizing a model with billions of parameters takes minutes to hours.
• Memory savings. A 4-bit model needs about a quarter of the weight memory of a 16-bit one.
• Wide support. Many runtimes and libraries can read GPTQ-style checkpoints.
• Faster decoding in memory-bound cases. When generation is limited by how fast weights can be read, smaller weights mean more tokens per second.

Limitations

• Calibration dependence. The method adapts the weights to the calibration inputs. If those inputs differ from real traffic, quality on real data can drop. This is a form of overfitting to the calibration set.
• Weight-only. The activations are still 16-bit, so compute-bound workloads, such as large-batch serving, may see little speedup. See SmoothQuant for the activation side.
• Kernel dependence. The speedup depends on the quality of the dequantizing kernel. A poor kernel can make a smaller model slower.
• Very low bits are hard. At 3 bits quality starts to slip, and at 2 bits it often drops sharply without extra tricks.
• Act-order trade-off. The best accuracy setting can reduce kernel speed.
• Sensitive layers. Some layers, such as embeddings and the final head, are more fragile and may need to stay in higher precision.
• No built-in uncertainty. The method gives a single rounded model with no estimate of how much was lost on any one input.

Why it works in plain words

Large networks are over-parameterized. Many weights carry redundant information, so there is slack to absorb small errors.

The input features of a layer are rarely independent. Strong correlations mean that one weight can stand in for another.

GPTQ exploits both facts. It spends the slack where the Hessian says it is cheap, and it repairs errors using the correlated weights that are still free.

A good picture is a team carrying a heavy table. If one person must let go, the others shift their grip. The load stays balanced, and nobody drops the table.

Why it can fail

The same facts explain the failures. If the calibration inputs do not excite a feature, the Hessian says that feature is unimportant. The method then rounds its weights harshly. If real traffic does excite that feature, the error shows up late and surprises you.

How it compares with alternatives

A few rules of thumb follow from this table.

• If you want the simplest route to a 4-bit model, GPTQ and AWQ are the usual choices.
• If compute-bound serving is the limit, look at methods that also quantize activations.
• If you want to fine-tune a compressed model, look at QLoRA and LoRA adapters.
• If you serve on CPUs or small devices, a block format such as GGUF is often easier.

Variants and extensions

Several follow-up ideas build on the same core.

• Grouped and mixed-bit schemes. Give more bits to sensitive layers and fewer to robust ones, under a fixed total budget.
• Vector and codebook quantization. Instead of rounding single weights, round small groups of weights to entries of a learned codebook. This can reach lower bit widths.
• Rotation-based preprocessing. Multiply weights and activations by an orthogonal matrix before quantizing so that outliers spread out. This pairs well with weight-only methods.
• Fine-tuning after quantization. Brief training of scales, zero points or adapters can recover lost quality at very low bit widths.
• Activation-aware extensions. Combine the error-compensation step with scaling ideas from AWQ or SmoothQuant.

These variants show that GPTQ is best seen as a building block. The inverse-Hessian update is a general tool for "round now, repair later".

When the method breaks down

• The Hessian is rank deficient, for example when the calibration set is tiny. Damping helps but does not fix it.
• A layer has extreme outliers in a few input channels. A single huge activation can dominate the Hessian and starve other channels.
• The runtime does not support the chosen group size or act-order setting.
• The model uses unusual layers that are not plain linear maps.

In each case the fix is to measure first, then change one setting at a time.

Role in tabular foundation models

Tabular foundation models learn from raw tables instead of text. A table has rows and columns, mixed types (numbers, categories, dates, short strings), missing values, and often no useful column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on large collections of tables, often synthetic ones, and then predict on a new table by in-context learning: the training rows of the new table are passed as context, and the model predicts the test rows in a single forward pass. See /model/architecture and /model/research for how this site describes such models.

Quantization matters here for concrete reasons.

Why the same weight layers can be compressed

Inside a tabular transformer, most parameters live in linear layers: attention projections that mix information across rows and across columns, and feed-forward blocks. These are exactly the layers GPTQ targets. A model with billions of parameters, trained on many table types, can be shrunk to fit a smaller GPU. That makes it cheaper to serve one model for many tables, for tasks such as classification, regression, imputation, anomaly detection and semantic sector labeling.

Calibration data must look like tables

The biggest practical change is the calibration set. For a language model you would use text. For a tabular model, the inputs to each layer are shaped by table structure: cell embeddings, column and row positions, a mask that marks missing values, and the split between context rows and query rows. The calibration set should therefore be real or synthetic tables that cover:

• numeric columns with different scales and skew,
• categorical columns with low and high cardinality,
• missing values at several rates,
• different table widths and row counts,
• the same mix of tasks you plan to serve.

If you calibrate on a narrow set of tables, the Hessian will describe only those, and unusual tables can degrade more than expected.

Numeric output is less forgiving

A language model can recover from a small wobble in one token. A regression head returns a number. A classification head returns probabilities that people may use for ranking or thresholds. Small systematic errors in weights can shift calibration of those probabilities. After quantization, measure not only accuracy but also calibration error, rank metrics and the error of numeric predictions.

Keep the input and output stages in higher precision

The layers that turn raw cells into embeddings, and the layers that turn hidden states into numbers, are small but sensitive. A common strategy is to quantize the large attention and feed-forward blocks and leave the encoders for numeric features, the missing-value handling and the final heads in 16-bit. This costs little memory and protects the parts that touch raw values.

In-context learning and the cache

Tabular in-context learning passes many context rows through the network. Memory use grows with the number of rows and columns because of attention and cached states. GPTQ compresses weights, not caches. For caches and vectors, a method such as TurboQuant is the better fit, and the two can be combined: GPTQ for weights, a cache method for stored keys and values. See also FlashAttention and mixed precision for related memory tools.

Continual learning and adapters

Some tabular systems keep learning from new tables over time, using ideas such as experience replay and EWC. A GPTQ model has frozen low-bit weights, so it is best treated as a fixed base. Learning is then done with small LoRA-style adapters in 16-bit on top of the quantized base, as in QLoRA. If the base is re-quantized after an update, the adapters must be checked again.

Imputation, anomalies and sector labels

• Imputation. Filling missing cells is a numeric task, so test the error of filled values on masked cells before and after quantization.
• Anomaly detection. Scores for rare rows depend on subtle differences. A model that is slightly flatter after quantization may blur the tails. Compare score rankings and false-alarm rates.
• Semantic and sector classification. When a model assigns a meaning or an industry label to a column or table, the task is closer to text. It tends to be more robust, but check rare classes, since low-bit models often lose rare cases first.

A cautious summary for tables

GPTQ is a good fit for the large linear layers of a tabular foundation model, as long as the calibration set looks like tables, the sensitive input and output layers stay in higher precision, and the evaluation looks at calibrated probabilities and numeric error, not only headline accuracy. Results on benchmarks should be rechecked after quantization; see /model/benchmarks for how this site frames evaluation.

Practical guidance and best practices

Choosing a configuration

• Start with 4 bits and a group size of 128. Move to 8 bits if quality is critical, or to smaller groups if you need more accuracy at the same bit width.
• Try act-order first and measure kernel speed. If speed suffers on your runtime, test without it.
• Use damping of about one percent and raise it if you see numerical errors.
• Use a few hundred calibration samples, then test whether doubling that number changes quality. If it does not, you have enough.

Choosing calibration data

• Match the domain of real traffic. For a tabular model, use tables, not text.
• Cover the range of sizes, types and missing rates you expect.
• Keep a separate evaluation set that was never used for calibration.
• Avoid duplicates and extreme outliers in the calibration set.

Choosing hardware and runtime

• Check which kernels your runtime has for the exact format. Optimized kernels exist in engines such as vLLM, ExLlamaV2 and TensorRT-based stacks, and support differs by GPU generation.
• Measure end-to-end latency at your real batch sizes. Gains at batch size one can vanish at large batches.
• Compare against a simple baseline, such as 8-bit weights, before committing.

A simple decision guide

Use these questions in order.

1. Does the full-precision model fit on your hardware with room to spare? If yes, you may not need quantization at all.
2. Is your bottleneck memory capacity or memory bandwidth? Weight-only methods such as GPTQ help both.
3. Is your bottleneck compute at large batch sizes? Consider 8-bit weights and activations.
4. Do you need to fine-tune afterward? Plan for adapters on a frozen quantized base.
5. Do you run on CPU or on a mixed device? Consider a block format and a runtime built for it.
6. Is the model small? Small models lose more from quantization, so test carefully or use 8 bits.

A short rollout plan

• Week one: quantize with default settings and compare quality on a held-out set.
• Week two: tune group size, act-order and calibration data, and benchmark the real runtime.
• Week three: shadow traffic. Run the quantized and full-precision models side by side and compare outputs.
• After launch: monitor quality metrics and keep the original model ready for rollback.

Validation checklist

1. Perplexity or task loss on held-out data.
2. Your real downstream metrics, including calibration and rank metrics where relevant.
3. Behavior on rare categories and edge cases.
4. Memory, latency and throughput on target hardware.
5. A diff of a sample of outputs between the 16-bit and quantized models.

Reading a quantization log

Most tools print a per-layer error while they run.

• The value is the squared output error of the layer after quantization.
• Errors usually grow in the middle layers and in the feed-forward blocks.
• A sudden spike in one layer often points to outliers or a bad calibration batch.
• A flat, tiny error everywhere may mean the layer matters little or the grid is wide enough.

Use the log to decide which layers to keep in higher precision.

Serving notes

• Warm up the kernels before measuring latency, since the first calls include setup cost.
• Test at the batch sizes, sequence lengths and row counts you really serve.
• Track memory over time, because fragmentation can erase the saving.
• Keep the same tokenizer or feature encoder as the original model.
• For tables, keep the same preprocessing, so that numeric scaling and missing-value masks match what the model saw in pre-training.

Operational habits

• Store the quantization config, calibration set description and library version next to the model.
• Keep the original 16-bit model so you can re-quantize with new settings.
• Re-run the validation after any library or kernel upgrade.

Common mistakes

• Calibrating on the wrong data. Using generic text to calibrate a tabular model, or one narrow domain for a general model, produces a model that looks fine on the calibration set and worse elsewhere.
• Judging only by perplexity. A small perplexity change can hide larger changes on specific tasks or rare classes.
• Ignoring kernel support. A format that your runtime handles through a slow fallback path will not give the speedup you expect.
• Quantizing everything. Embeddings, normalization layers and output heads are often better left in higher precision.
• Too few or too many samples. Very few samples give a noisy Hessian. Very many cost time with little gain.
• Forgetting damping. Without it, the inverse can fail or become unstable.
• Assuming 3 bits or 2 bits behave like 4 bits. Quality can fall sharply at the lowest widths.
• Comparing against the wrong baseline. Compare with a well-chosen 8-bit or round-to-nearest baseline, not only with the 16-bit model.
• Quantizing twice. Applying quantization on top of an already quantized checkpoint compounds the error.
• Skipping a regression test after updates. Library changes can alter numerical results.

Frequently asked questions

What does GPTQ stand for and what does it do?

GPTQ is the name of a post-training quantization algorithm first aimed at generative pre-trained transformers. It converts the weights of a trained model from 16-bit numbers to low-bit integers, usually 4 bits, while correcting rounding errors layer by layer so the model output stays close to the original.

Does GPTQ need to retrain the model?

No. GPTQ is a post-training method. It needs a small calibration set to measure how the inputs of each layer behave, but it does not run gradient descent over a training dataset. This is why it can finish in minutes or hours on one GPU.

How much smaller does GPTQ make a model?

At 4 bits per weight the weight storage is roughly a quarter of the 16-bit size, plus a small overhead for scales and zero points. Total memory use also includes activations and caches, which GPTQ does not shrink, so the overall saving at run time is smaller than four times.

Is GPTQ better than AWQ?

Neither is always better. GPTQ uses second-order information and error compensation, while AWQ protects the weight channels that matter most for the activations and avoids reconstruction. Results depend on the model, bit width, group size and kernel support, so the best approach is to test both on your own task.

Does GPTQ make inference faster?

It can. When generation is limited by memory bandwidth, reading smaller weights speeds up decoding. When the workload is compute-bound, for example with large batches, a weight-only method may help little, and a method that also quantizes activations such as SmoothQuant may be a better choice.

What is the role of the calibration set?

The calibration set provides sample inputs for each layer, from which GPTQ builds its Hessian. The algorithm then adapts the weights to those inputs. If the set is not representative of real use, quality on real data can be lower than the numbers measured on the calibration set suggest.

Can GPTQ be used on models that learn from tables?

Yes, in principle. The large linear layers of a tabular transformer are good candidates. The calibration data should be tables with realistic types, scales and missing values, and the input encoders and output heads are often better kept in higher precision. Always recheck calibration, numeric error and rare-class behavior afterward.

Can I fine-tune a GPTQ model?

You can train small adapters on top of a quantized base, which is the idea behind QLoRA and LoRA. The base weights stay frozen and low-bit, and only the adapters are updated in higher precision. Full fine-tuning of the quantized integer weights is not the normal workflow.

What bit width should I choose?

Four bits with a group size of 128 is the common starting point because it balances size and quality. Eight bits is safer when accuracy is critical. Three bits and below usually need more care, smaller groups or additional techniques, and they should be validated more strictly.

How is GPTQ related to pruning?

The method descends from pruning research that used the inverse Hessian to decide which weights to remove and how to adjust the rest. GPTQ uses the same compensation idea, but instead of setting weights to zero it rounds them to a coarse grid and corrects the remaining weights for the rounding error.

Glossary

• Activation. The output of a layer, passed to the next layer.
• Act-order. A setting that quantizes columns with larger activation energy first.
• Calibration set. A small sample of inputs used to measure layer statistics.
• Cholesky decomposition. A way to factor a symmetric positive definite matrix into a triangular matrix and its transpose.
• Damping. A small value added to the Hessian diagonal for numerical stability.
• Dequantization. Converting integer codes back to real numbers using scale and zero point.
• Group size. The number of weights that share one scale and zero point.
• Hessian. A matrix of second derivatives that describes curvature; here proportional to the input covariance.
• In-context learning. Predicting on new data by reading examples in the input, without updating weights.
• Kernel. A low-level GPU routine that performs an operation such as a matrix multiply.
• Packing. Storing several low-bit values inside one wider integer.
• Perplexity. A language-model quality score; lower is better.
• Post-training quantization. Reducing numeric precision after training is finished.
• Round-to-nearest. The simplest quantization, which rounds each value independently.
• Scale. The step size between neighboring quantized values.
• Weight-only quantization. Storing weights in low precision while keeping activations at higher precision.
• Zero point. The integer that represents real zero on an asymmetric grid.

Summary

GPTQ compresses a trained network to 4 bits or fewer by quantizing each layer one column at a time and spreading each rounding error across the weights that remain, guided by an inverse Hessian built from a small calibration set. It needs no retraining, runs on a single GPU, and keeps quality close to the original for many large models.

Its main limits are dependence on calibration data, a weight-only focus, kernel support and sensitivity at very low bit widths. For tabular foundation models, it can shrink the large attention and feed-forward layers if calibration uses realistic tables and the input and output stages stay precise. Pair it with AWQ, SmoothQuant or TurboQuant as needed, and validate on the metrics that matter for your task, as described in /model/benchmarks.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: