TurboQuant

Overview

TurboQuant is a quantization method from Google Research, published at ICLR 2026, for compressing high-dimensional vectors to a few bits each with very little error. It combines two ingredients. The first, PolarQuant, rotates and re-expresses a vector so that it can be rounded efficiently with almost no extra bookkeeping. The second, a residual step based on the quantized Johnson-Lindenstrauss transform, called QJL, spends a single bit per value to remove the bias that rounding leaves in inner products.

The method works online and without training. It does not learn a codebook from data, so a new vector can be compressed the moment it appears. That makes it well suited to things that grow while a model runs, such as the key-value cache of a transformer, and to vector search. It sits beside GPTQ, AWQ and SmoothQuant, which mainly compress weights, and it addresses a different part of the memory problem. This article explains the ideas in plain terms, shows a worked example, and discusses how the method relates to models that learn from raw tables. Where the exact details of the paper matter, this article speaks in general terms, and you should consult the original publication.

What it is

TurboQuant is a vector quantizer. Given a vector of many numbers, it returns a short code of a few bits per number. From that code you can recover an approximate vector, or you can estimate an inner product with another vector directly.

Its main properties:

• Online. Each vector is compressed on its own, with no need to see a dataset first.
• Data-oblivious. It does not train a codebook or cluster centers, so there is no training or tuning phase.
• Low bit width. It targets very few bits per coordinate, including values close to one bit.
• Low distortion. The error is close to what theory says is the best possible for a given number of bits.
• Unbiased inner products. The residual correction makes estimated dot products correct on average, which matters for attention and for search.
• Accelerator friendly. The steps are simple linear algebra and rounding.

An everyday analogy: you want to mail a large painting in a small envelope. A careless approach is to cut it into pieces and hope. A good approach is to first turn the painting so that its detail is spread evenly across the canvas, then shrink it in a regular way, and finally add a tiny note that corrects the main distortion. PolarQuant is the careful turning and shrinking. The QJL step is the small correcting note.

What problem it targets

Large transformers store, for every token already seen, a key vector and a value vector in every layer. This is the key-value cache. For long inputs and many simultaneous users, the cache can take more memory than the weights. Vector databases face a similar problem: billions of embedding vectors must be stored and searched. Both need compression that is fast, simple and accurate enough for dot products.

What it is not

• It is not a weight-only compression recipe for model files, although the ideas can be used for weights.
• It is not a learned method like product quantization, which fits a codebook to a dataset.
• It is not lossless. It trades a small, well-understood error for large memory savings.
• It is not a replacement for good model design. It reduces memory, not computation per se.

Where it came from

Vector quantization is old. Classical methods such as k-means codebooks and product quantization learn a set of representative vectors from data. They work well when the data distribution is fixed and known, but they require training, they are slow to update, and they behave poorly when the data shifts.

Several lines of research feed into TurboQuant.

• Random projections. The Johnson-Lindenstrauss lemma, from the 1980s, says that a random projection to a lower dimension approximately preserves distances. Later work used random projections and sign bits for hashing and similarity search.
• Random rotations. Multiplying a vector by a random orthogonal matrix spreads its energy across coordinates. After rotation, the coordinates have a predictable distribution, so a fixed scalar quantizer can be used without learning anything from data.
• Information theory. Rate-distortion theory gives lower bounds on the error achievable with a given number of bits. Recent work aimed to match those bounds with practical algorithms.
• KV cache compression. As context windows grew, researchers proposed ways to quantize keys and values, often with per-block scale factors that cost extra memory.
• QJL. A quantized Johnson-Lindenstrauss transform that keeps only the sign of random projections and gives an unbiased estimator of inner products through an asymmetric estimator.
• PolarQuant. A method that converts pairs of coordinates into radius and angle, recursively, so that angles have a concentrated distribution and no per-block normalization constants are required.

TurboQuant combines these threads. It uses a randomized transform to make the coordinates well behaved, applies an optimal scalar quantizer for the resulting distribution, and then adds a one-bit residual step to correct the bias in inner products. The publication reported near-optimal distortion for a given bit budget, and results for key-value cache compression and nearest-neighbor search. For the research context of this site, see /model/research.

Other tools in this family of ideas include weight methods such as GPTQ, AWQ and SmoothQuant, and practical formats such as GGUF and bitsandbytes. They share the goal of fewer bits per number, but they operate on different objects.

How it works

Step 1: Why vectors are hard to round

Take a vector with many coordinates. If you round each coordinate to a few bits, two problems appear.

First, vectors in a cache have coordinates with very different ranges. A few large coordinates force a wide grid, and the small coordinates are rounded to nothing.

Second, most methods store extra numbers, such as a scale and a zero point for each block of values. At very low bit widths these extra numbers can cost almost as much as the codes themselves.

TurboQuant is designed to remove both problems.

Step 2: Randomly rotate the vector

Multiply the vector by a random orthogonal matrix. A rotation keeps the length and all angles. It changes only which coordinates carry the information.

After a random rotation, the energy of the vector is spread nearly evenly. Each coordinate follows a known, bell-shaped distribution that depends only on the dimension and the length of the vector. No coordinate is much larger than the others.

Analogy: pour a bag of sand that is heaped in one corner of a tray, then shake the tray until the sand lies flat. Now any strip of the tray holds about the same amount, so one measuring cup fits all strips.

Step 3: Quantize each coordinate with a fixed optimal grid

Because the distribution after rotation is known in advance, the best rounding grid for it can be computed once, offline, and reused for every vector. This is a classical scalar quantization problem: choose levels that minimize the average squared error for a given distribution and number of bits.

No data is needed to build the grid. That is the sense in which the method is data-oblivious.

Step 4: Where PolarQuant fits

PolarQuant offers another way to reach the same goal. Group the coordinates into pairs and treat each pair as a point in the plane. Express it as a radius and an angle. Combine the radii of pairs into new pairs and repeat, recursively. The result is one overall radius and a tree of angles.

The angles, after a random rotation, have a concentrated and known distribution. They can be quantized with a fixed grid. The radius is one number and can be stored at higher precision. Because the angle distribution is known, there is no need to store a scale or zero point for each small block. That removes the overhead of normalization constants.

Analogy: describing a location by a heading and a distance. All headings look alike, so a fixed compass with a few marks works for everyone, and only the distance varies.

Step 5: Understand the leftover error

The first stage gives a good reconstruction in the sense of mean squared error. But attention and search rely on inner products, and an estimator based on the reconstruction can be biased. Shrinking and rounding tend to pull values toward zero, so inner products come out slightly too small on average.

A bias is different from noise. Noise averages out over many terms. A bias adds up in one direction and can distort attention scores systematically.

Step 6: Compress the residual with QJL

The residual is the difference between the original vector and its first-stage reconstruction. TurboQuant projects the residual with a random matrix and keeps only the sign of each projected value. That is one bit per value.

The sign bits, together with the residual's length, support an asymmetric estimator. When you later compute an inner product with a full-precision query, the estimator is unbiased: its average equals the true inner product. The cost is one extra bit per value and a small constant.

Analogy: after a first rough drawing of a route, you add a single yes-or-no hint at each junction that tells which way the route drifted. The hints are crude, but they cancel the systematic drift.

Step 7: Combine the two stages

The code for a vector therefore holds:

1. The low-bit codes from the first stage (rotation plus scalar or polar quantization).
2. The sign bits of the projected residual.
3. A small number of scalars, such as the vector length and the residual length.

To estimate an inner product with a query, combine the first-stage reconstruction with the correction from the sign bits. The total error is close to the theoretical minimum for the bit budget, and the estimate is unbiased.

Step 8: Use it online

Because nothing is learned from data, the same rotation matrix and the same grids work for every vector. When a new token arrives, its key and value are compressed immediately and appended to the cache. There is no batch step, no clustering and no re-training.

Step 9: Use it in attention or search

In attention, scores are inner products between the query and the cached keys. With the unbiased estimator, scores can be computed from the compressed cache. Values can be reconstructed from the first-stage codes when needed. In vector search, the same estimator ranks stored vectors against a query.

Why random rotation is cheap enough

A dense random rotation costs a quadratic number of operations in the dimension. Practical implementations use structured transforms, such as fast Hadamard-style transforms with random signs, that cost far less and still give the needed spreading effect. The rotation is applied once per vector.

A note on exact details

The paper specifies the precise quantizers, bit allocations between stages and the constants in the estimator. This article describes the structure in general terms. For exact formulas, bit widths and measured results, consult the original publication.

Key concepts and terms

A worked example

A numeric walk-through in two dimensions

Take a vector v = (3, 4). Its length is 5.

In polar form it has radius 5 and angle about 53.13 degrees. Suppose we quantize the angle with 2 bits, which gives four levels over a quarter turn: 0, 30, 60 and 90 degrees. The nearest level is 60 degrees. The reconstruction is 5 times (cos 60, sin 60) = (2.5, 4.33).

The residual is (3, 4) minus (2.5, 4.33) = (0.5, -0.33).

Now compare inner products with a query q = (1, 0).

• True: 3.
• Using the reconstruction only: 2.5. This is lower than the truth.

A one-bit residual step projects the residual on random directions and keeps the signs. Averaged over random choices of directions, the correction term reconstructs the missing 0.5 on average. Any single estimate is noisy, but the long-run average is exact. In a real vector with hundreds of coordinates and many cached tokens, the noise averages out and the bias is gone.

This is a toy case. Real vectors have many coordinates, and the recursive pairing and the random rotation make the grids work uniformly.

A Python sketch of the structure

The code below shows the structure of a rotation, a scalar quantizer and a one-bit residual sketch. It is a teaching aid with a dense random rotation, not an optimized or exact reproduction of the published algorithm.

import numpy as np

rng = np.random.default_rng(0)

def random_rotation(d):
    a = rng.standard_normal((d, d))
    q, r = np.linalg.qr(a)
    return q * np.sign(np.diag(r))

def scalar_quantize(x, bits, clip=3.0):
    # uniform grid on [-clip, clip] after scaling by the coordinate std
    levels = 2 ** bits
    step = 2 * clip / levels
    idx = np.clip(np.floor((x + clip) / step), 0, levels - 1)
    return -clip + (idx + 0.5) * step

class SketchQuantizer:
    def __init__(self, d, bits=3, m=None):
        self.d, self.bits = d, bits
        self.R = random_rotation(d)                  # rotation
        self.m = m or d
        self.S = rng.standard_normal((self.m, d))    # projection for residual

    def encode(self, v):
        norm = np.linalg.norm(v)
        u = self.R @ (v / norm) * np.sqrt(self.d)    # unit-variance coordinates
        q = scalar_quantize(u, self.bits)
        resid = u - q
        signs = np.sign(self.S @ resid)
        return norm, q, signs, np.linalg.norm(resid)

    def inner(self, code, query):
        norm, q, signs, rnorm = code
        qu = self.R @ query * 1.0                    # rotated query (full precision)
        base = (q / np.sqrt(self.d)) @ qu
        # QJL-style unbiased estimate of <resid, query_rot>
        proj = self.S @ qu
        corr = np.sqrt(np.pi / 2) / self.m * rnorm * np.sum(signs * proj) / np.sqrt(self.d)
        return norm * (base + corr)

d = 128
tq = SketchQuantizer(d, bits=3)
v, q = rng.standard_normal(d), rng.standard_normal(d)
print("true", v @ q, "estimate", tq.inner(tq.encode(v), q))

Run it for many random pairs and compare the average error of the estimate with and without the correction term. The correction removes the systematic offset, though each estimate stays noisy.

Using it in practice

Most users will meet this method through an inference engine or a library that exposes a cache-quantization option, not by writing the algorithm. A typical flow:

1. Choose the target bit width for keys and values.
2. Enable cache compression in the serving engine or library.
3. Run a quality check on long-context tasks.
4. Measure memory per request and tokens per second.

Engines such as vLLM and stacks built on TensorRT or PyTorch expose cache quantization options in various forms. Support for any specific algorithm depends on the version, so check the documentation of your engine.

Memory arithmetic for a cache

A rough formula for a cache in 16-bit:

• Bytes = 2 (keys and values) times layers times heads times head dimension times tokens times 2 bytes.

Compressing from 16 bits to about 3 bits per number cuts the cache by more than five times. More tokens fit on the same GPU, or more users share it. The exact figure depends on how many bits the residual step and the scalars add.

Checking the result

• Compare outputs with the 16-bit cache on long prompts.
• Test retrieval-style tasks, in which one detail deep in the context matters.
• Compare memory per request and total throughput.
• Test at the context lengths you really serve.

Strengths and limitations

Strengths

• No training. No calibration set or codebook training is required.
• Online. New vectors are compressed immediately, which suits growing caches.
• Near-optimal error. Distortion is close to the theoretical limit for the bit budget.
• Unbiased inner products. The residual step keeps attention scores and similarity rankings fair.
• Little overhead. The polar view avoids per-block constants, so low bit widths keep their benefit.
• General. It applies to caches, embeddings and, in principle, any vectors.
• Simple operations. Rotation, rounding and sign bits suit accelerators.

Limitations

• Targets vectors, not model files. It does not replace weight methods for shrinking a checkpoint.
• Kernel support matters. Without fused kernels, compression and decompression can cost time.
• Randomness. The estimator is unbiased but not exact, so single estimates are noisy. Fixed seeds are needed for reproducibility.
• Task sensitivity. Tasks that need exact recall of a detail may be more sensitive than average-quality metrics suggest.
• Not free of cost. Rotation and residual projection add compute, although small.
• Maturity. As a recent method, tooling and independent comparisons are still developing.
• Edge cases. Vectors with extreme structure, such as very sparse ones, may need testing.

Quick facts

• Typical target: key-value caches and embedding stores.
• Typical budget: a few bits per number, with a small extra for lengths and sign bits.
• Typical setup: no calibration, no codebook, a fixed seed for the random transforms.
• Typical gain: several times less memory for the stored state.
• Typical risk: tasks that need exact recall of one detail in a long context.

These are general starting points, not guarantees. Measure on your own workload.

Why it works in plain words

Imagine packing many uneven boxes into a truck. If you first reshape every box into a similar form, one standard packing rule works for all of them. That is the random rotation.

Then you pack each box with that rule. Some small damage is unavoidable, and it has a direction: items tend to shrink a little. A single yes-or-no note per item, the sign bit, lets the receiver correct that drift on average.

The result is a compact load that behaves fairly when you later ask how similar two items are.

Why it can fail

• The rotation or projection seeds differ between encoding and estimating.
• The query is also compressed, which breaks the asymmetric estimator.
• The kernels are not fused, so extra memory traffic cancels the gain.
• The task depends on a single exact detail that noise can blur.

Where TurboQuant is a good choice

• Long-context serving where the cache dominates memory.
• Vector search over very large collections of embeddings.
• Systems that compress vectors as they arrive and cannot train a codebook.
• Pipelines where unbiased similarity estimates matter.

How it compares with alternatives

TurboQuant versus weight quantization

Weight methods shrink the checkpoint and speed up memory-bound decoding. TurboQuant shrinks the state that grows during inference. They address different memory pools and can be combined: for example, 4-bit weights from AWQ or GPTQ, together with a compressed cache.

TurboQuant versus product quantization

Product quantization learns codebooks from a dataset and can reach strong compression on a fixed collection. It is hard to update when data changes and costs training time. TurboQuant needs no training and handles new vectors immediately, at the price of not exploiting dataset-specific structure.

TurboQuant versus simple cache quantization

Simple per-block rounding stores a scale and zero point per block. At 3 or 4 bits those extras eat much of the saving, and the rounding bias can distort attention. TurboQuant removes most of the overhead and corrects the bias.

TurboQuant versus eviction methods

Some systems save cache memory by dropping tokens judged unimportant. Eviction loses information entirely. Quantization keeps all tokens at lower precision. The two can be combined, though each adds risk to retrieval tasks.

Role in tabular foundation models

Tabular foundation models learn from raw tables, not text. A table has rows and columns, mixed types, missing values, and often missing or meaningless column names. Models such as TabPFN, TabICL and TabSTAR are pre-trained on many tables, often synthetic ones, and then predict for a new table by in-context learning. The training rows of the new table are given as context, and the model predicts the test rows in a forward pass. See /model/architecture for how this site describes such models.

TurboQuant is relevant in several concrete ways.

The context is a table, and tables are big

In language models, the cache grows with the number of tokens. In a tabular model that uses in-context learning, the analog is the context table: every context row, and often every cell, has to be represented by hidden states or cached keys and values. A table with many rows and columns can produce a very large state. Compressing that state with an online, training-free quantizer lets a single GPU handle larger context tables or more tables at once.

Reusing the context across many queries

A common serving pattern is to fix a labeled context table and then predict many query rows against it. If the model supports caching the context, the cache can be stored once and reused for every query. Compressing that stored state with TurboQuant means that more context rows fit in memory, and the cost of reuse falls. This benefits tasks such as scoring many new records against a reference table.

Embeddings of rows, columns and cells

Tabular systems often produce embeddings: a vector for each row, column or cell. These are used for similarity search, for deduplication, for finding near-duplicate records and for retrieval of relevant context rows. Those are inner-product problems, exactly the setting in which unbiased estimates matter. TurboQuant can compress such embedding stores without training a codebook that would be tied to one table.

Missing values and mixed types

Tables have missing values and mixed types, which the model typically handles with masks and type-specific encoders before the transformer. TurboQuant acts after those encoders on internal vectors, so it does not need to know about types or masks. Still, test that vectors from rows with many missing values, which may have unusual structure, are compressed well.

No column names

When column names are missing, the model relies on values and positions. The relationships between columns are encoded in its hidden states. Compression must preserve inner products between those states, which is what the unbiased estimator targets.

Pre-training on synthetic tables

Models pre-trained on synthetic tables see broad distributions. Because TurboQuant is data-oblivious, it does not tie itself to a particular pre-training distribution. A codebook learned on one family of tables might not suit another. A random rotation and a fixed grid do not have that problem.

Imputation and anomaly detection

• Imputation. Filling a missing cell often uses attention over similar rows. Compressed caches change attention scores slightly, so check imputation error on masked cells. See imputation networks.
• Anomaly detection. Scores for unusual rows rely on subtle differences from normal rows. A biased compressor could hide or exaggerate them, which is why the unbiased estimator is attractive. Compare ranking quality and false alarm rate. See anomaly detection.
• Semantic and sector classification. Assigning a meaning or an industry label to a column or table uses embedding similarity, so compressed embeddings should be tested on rare classes.

Combine with weight quantization

A tabular foundation model can use GPTQ or AWQ for weights and TurboQuant for the stored context state. Keep numeric encoders and output heads in higher precision unless tests show otherwise. Check regression error, probability calibration and rank metrics after each change.

Continual learning

When a model keeps learning from new tables with adapters such as LoRA, the cache must be recomputed after updates, because hidden states change. Methods such as experience replay and EWC concern the learning step, not the cache, but a stable evaluation after each update is still needed. Evaluate with the approach described at /model/benchmarks.

Related memory tools

For attention memory, see FlashAttention. For number formats, see mixed precision.

Practical guidance and best practices

Choose the bit width

• Start with a moderate budget, such as 4 bits per number, then lower it step by step.
• Evaluate each step on your real tasks, especially those that need exact recall.
• Treat the lowest settings as experimental until you have tested them.

Choose what to compress

• Keys usually matter more for attention accuracy than values, so you may allocate bits unevenly.
• Compress old context more aggressively than recent context if your engine allows it.
• For tables, compress the context state, and keep the query and output path precise.

Make it reproducible

• Fix the seeds for the rotation and the projection.
• Store the seeds and library version with the deployment.
• Use the same transform for encoding and for estimating.

Choose the runtime

• Check that your engine supports fused kernels for the chosen scheme.
• Benchmark at real context lengths and batch sizes.
• Compare memory, latency and quality against a plain 8-bit cache.
• Consider vLLM, TensorRT-based stacks or ONNX Runtime depending on your hardware.

Validate

1. Long-context quality on tasks like retrieval, summarization or, for tables, prediction accuracy as the number of context rows grows.
2. Calibration of probabilities and numeric error for table tasks.
3. Rank metrics for anomaly and similarity tasks.
4. Memory per request and throughput.
5. Output comparison against an uncompressed cache.

A decision guide

1. Is the cache or embedding store your main memory cost? Consider TurboQuant.
2. Is the weight file your main cost? Consider AWQ or GPTQ.
3. Is compute at large batch your limit? Consider SmoothQuant.
4. Do you need local CPU inference? Consider GGUF with llama.cpp.
5. Do you have a fixed, static vector set? A trained codebook may beat a data-oblivious method.

Operational habits

• Keep an uncompressed baseline path for debugging.
• Monitor quality metrics after launch.
• Re-run validation after every engine upgrade.
• Log context lengths and compression settings with each request in test environments.

Common mistakes

• Treating it as a weight compressor. The method targets vectors and caches. Using it to shrink a checkpoint is a different job.
• Testing only short prompts. Cache errors accumulate over long contexts.
• Judging only by average quality. Retrieval of a single detail can fail while averages look fine.
• Ignoring bias. A compressor that rounds inner products down can shift attention. Always check that the estimator is unbiased in your implementation.
• Forgetting seeds. Different seeds for encode and decode break the estimator.
• Skipping the baseline. Compare with a simple 8-bit cache before adopting a more complex scheme.
• Compressing the query path. Keep queries in full precision, since the estimator is asymmetric.
• Ignoring kernel cost. If compression is not fused, latency may rise.
• Assuming exact numbers from a summary. Check the original publication for exact bit allocations and results.
• Compressing without retesting after model updates. Hidden state statistics change when the model changes.

Frequently asked questions

What is TurboQuant?

TurboQuant is a quantization method from Google Research, published at ICLR 2026, that compresses high-dimensional vectors to a few bits per number with very little error. It combines PolarQuant, a polar-coordinate style quantizer, with a one-bit QJL residual step that removes bias in inner products.

What do PolarQuant and QJL do?

PolarQuant expresses coordinates as a radius and angles with a concentrated distribution, so a fixed grid works and no per-block constants need storing. QJL projects the leftover error with random directions and keeps only the signs, which lets the system estimate inner products without bias.

Does TurboQuant need training or calibration data?

No. It is data-oblivious and online. The rotation and the grids are fixed in advance and do not depend on a dataset, so a new vector can be compressed as soon as it appears. This distinguishes it from learned codebook methods and from calibration-based weight quantizers.

Is TurboQuant used to shrink model weights?

Its main targets are vectors and caches, such as the key-value cache of a transformer and embedding stores for search. Weight compression is usually handled by methods like GPTQ and AWQ. The two kinds of method address different memory pools and can be used together.

Why does unbiasedness matter?

Attention and similarity search depend on inner products. A biased compressor makes scores consistently too low or too high, which shifts which items get attention or rank first. Noise averages out across many terms, but bias accumulates, so removing bias is more important than shaving a little noise.

How much memory can it save?

The saving depends on the bit width. Moving from 16-bit numbers to about 3 or 4 bits per number cuts the stored state by roughly four to five times, minus small overheads for lengths and sign bits. Exact numbers depend on the scheme and settings, so measure them on your own workload.

Can it be used with tabular foundation models?

Yes, in principle. Tabular models that use in-context learning hold a large state for the context table, and they produce embeddings of rows and columns. Compressing that state or those embeddings with an online, training-free quantizer saves memory. Test numeric error, calibration, ranking quality and rare-class behavior afterward.

How is it different from product quantization?

Product quantization learns codebooks from a dataset, which can compress a fixed collection well but takes training and is hard to update. TurboQuant uses a random rotation and fixed grids, so it needs no training and handles new vectors instantly, though it does not exploit dataset-specific structure.

Does it make inference faster?

It can, when memory traffic for the cache is the bottleneck, since less data is read per token. It also lets more requests share a GPU. Without fused kernels, extra compute for rotation and decoding may offset the gain, so measure end-to-end latency and throughput.

Where can I find the exact algorithm?

The exact quantizers, bit allocations, constants and results are in the original publication. This article explains the ideas in general terms and does not reproduce every detail. When you implement it, follow the paper and verify unbiasedness and distortion on your own data.

Adoption checklist

Before turning TurboQuant-style compression on for a real workload, it helps to walk through a short checklist. Each item is a question with a concrete way to answer it.

Decide what you are compressing

• Is the target the key-value cache, an embedding index, or a table of row and column vectors?
• How many vectors are stored, and how long do they live?
• Is memory capacity or memory bandwidth the real bottleneck?

Choose the bit budget

• Start from the largest budget that fits in memory, then lower it step by step.
• Measure quality at each step on the task you care about, not on a proxy.
• Keep a full-precision reference run so every comparison has a fixed baseline.

Check the estimator

• Confirm that inner products are estimated without systematic bias.
• Compare the spread of errors, not only the average error.
• Test long contexts separately, because small errors can add up over many tokens.

Check the kernels

• Make sure rotation, packing and dequantization are fused where possible.
• Profile memory traffic before and after, since savings only matter if reads actually shrink.
• Confirm that batching and caching behave the same with compressed state.

Plan for rollback

• Keep the full-precision path available behind a switch.
• Log quality metrics continuously after rollout.
• Re-test whenever the model, the data or the hardware changes.

Following this list turns compression from a one-time experiment into a routine, repeatable step that can be audited later.

Glossary

• Asymmetric estimator. An estimator that treats the stored vector and the query differently, with the query kept in full precision.
• Bias. A consistent error in one direction.
• Data-oblivious. Not depending on a sample of data.
• Distortion. The average error introduced by compression.
• Embedding. A vector that represents an item such as a row, column or document.
• Inner product. The sum of products of matching coordinates, also called a dot product.
• Johnson-Lindenstrauss transform. A random projection that approximately preserves distances.
• Key-value cache. Stored key and value vectors of past tokens.
• Online algorithm. One that processes each item as it arrives.
• Orthogonal matrix. A matrix that rotates or reflects vectors without changing lengths.
• PolarQuant. A quantizer based on radius and angle representation.
• QJL. Quantized Johnson-Lindenstrauss, which keeps only signs of random projections.
• Rate-distortion. The trade-off between bits used and error.
• Residual. The part of a vector left over after a first approximation.
• Scalar quantizer. A quantizer that rounds each number separately.
• Unbiased estimator. An estimate whose average equals the true value.

Summary

TurboQuant compresses vectors and caches to a few bits per number by rotating each vector so that its coordinates are well behaved, quantizing with a fixed grid or polar representation, and then spending one bit per value on a QJL residual step that makes inner product estimates unbiased. It needs no training, works online, and keeps error close to theoretical limits.

It complements weight methods such as GPTQ, AWQ and SmoothQuant. For tabular foundation models, it is most useful for the large context state and for row, column and cell embeddings, provided that numeric error, calibration, ranking quality and rare classes are checked afterward. See /model/architecture and /model/benchmarks for model and evaluation context, and consult the original publication for exact details.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: