ExLlamaV2

Overview

ExLlamaV2 is an inference library for running quantized large language models quickly on a single GPU, especially the kind of graphics card found in a desktop computer. It comes with its own quantization format, called EXL2, which can spend a different number of bits on different parts of a model so that the average size lands at a chosen target, for example somewhere between two and eight bits per weight. Custom GPU kernels read these packed weights and run the model with low memory use and high speed.

The library matters because it makes large models practical on modest hardware. A model that needs far more memory than a consumer card offers in 16-bit form can fit when stored in a low-bit format, and the kernels keep generation fast. ExLlamaV2 sits in the same family as GPTQ, AWQ and GGUF with llama.cpp, and it is usually compared with serving engines such as vLLM. Its sweet spot is one user, one GPU and high single-stream speed, rather than serving thousands of requests at once.

What it is

ExLlamaV2 has two linked parts.

• A quantization format and tool chain. The format is EXL2. The tool measures how sensitive each layer is to rounding, chooses bit widths and writes a compact model file.
• An inference engine. It loads EXL2 models (and several other formats), runs the transformer on a GPU with hand-written kernels, manages the key-value cache and generates tokens.

It is written mostly in Python with CUDA kernels underneath, and it targets NVIDIA GPUs, with support for some other GPU families depending on version. The library can be used directly from Python, through a small command line interface, or behind servers and chat front ends that embed it as a model loader.

The idea in one picture

A language model is a stack of layers made of large matrices of numbers called weights. Storing each weight in 16 bits is accurate but heavy. Quantization stores each weight with fewer bits and a scale factor, trading a little accuracy for much less memory. Not all weights are equally important. Some layers tolerate rough rounding. Others break easily. EXL2 gives more bits to the sensitive parts and fewer to the tolerant parts, so that the average stays low while quality stays high.

Analogy: packing a suitcase with a weight limit. Fragile items get thick padding. Sturdy items get none. The total weight stays under the limit, and the fragile items arrive intact.

What it is not

• It is not a training library. It runs models that are already trained.
• It is not a general serving platform with scheduling for many users. Other engines are built for that.
• It is not a model. It is the software that runs models.
• It is not tied to one model family. It supports a range of popular transformer architectures, with support varying by version.

Where it came from

ExLlamaV2 grew out of a hobbyist and open-source effort to run large language models on consumer graphics cards. An earlier project in the same line focused on running 4-bit GPTQ models fast on a single GPU. The second version rewrote the engine and added a new quantization format so that users could choose almost any average bit width instead of a few fixed ones.

Background that made it possible

• Post-training quantization methods such as GPTQ, published in 2022, showed that large models could be quantized to 4 bits or fewer with small quality loss, using a small calibration dataset and second-order information about the weights.
• Activation-aware ideas, as in AWQ, showed that a small share of weights matter more than the rest.
• Open model releases created demand for running large models locally.
• Consumer GPUs gained enough memory bandwidth to make low-bit inference fast, as long as the kernels were written well.

Evolution

Over time the library added several features: a dynamic generator that can batch and reuse cache pages, support for speculative decoding with a small draft model, quantized key-value caches, and support for more model families. Server projects and chat front ends adopted it as a backend. A newer generation of the library exists as a separate project with its own quantization format. This article is about the second-generation library and its EXL2 format, and it avoids claims about exact version dates or numbers.

How it works

This section walks through what happens from quantizing a model to generating text, using everyday analogies.

Step 1: Start from a trained model

The input is a full-precision model, usually stored in 16-bit floating point. It has many layers, each with attention matrices and feed-forward matrices.

Step 2: Prepare calibration data

Quantization is guided by a calibration dataset: a modest amount of representative text. The quantizer runs it through the model to see how activations behave. The choice of calibration text matters, since the quantizer optimises rounding for the kind of inputs it sees.

Analogy: a tailor measures a customer while they wear typical clothes, not while they stand in a winter coat.

Step 3: Measure sensitivity

The EXL2 converter runs a measurement pass. For each module, it tries several quantization settings and records how much error each introduces. This builds a table of "cost in error" versus "cost in bits" for every part of the model.

Analogy: a mechanic tests each part of an engine to learn which ones can be lighter without hurting performance.

Step 4: Choose bit widths under a budget

You give a target average, called bits per weight. The converter then picks a mix of settings that minimises total error while meeting the average. Sensitive modules get higher precision, such as 6 or 8 bits. Tolerant modules get 3 or 4 bits, or even 2. The result might average 4.25 bits per weight overall, even though no single layer uses exactly that.

Analogy: a budget planner who spends more on rent and less on snacks to stay within a monthly limit.

Step 5: Quantize with error compensation

Within each matrix, weights are quantized in groups with scale values. As in GPTQ-style methods, the process goes column by column and updates the remaining weights to compensate for the error just introduced. This keeps the output of each layer close to the original on the calibration data.

Step 6: Write the compact model

The quantized weights, scales and the per-module settings are saved in a packed file format. The file is far smaller than the original, which cuts disk use, load time and memory.

Step 7: Load and place on the GPU

At run time, the library reads the file and puts layers on the GPU. If the model is too big for one card, layers can be split across several cards, though the library is best known for the single-card case. The loader sets up the key-value cache, which stores past attention keys and values so that new tokens do not recompute them.

Step 8: Run the kernels

The custom kernels read packed low-bit weights and multiply them by activations directly, unpacking on the fly inside the GPU. Because generation of one token at a time is limited mostly by memory bandwidth, smaller weights mean fewer bytes to read, and tokens come out faster.

Analogy: a courier carrying a lighter parcel can make more trips per hour.

Step 9: Generate tokens

The generation loop is the standard autoregressive one:

1. Run the prompt through the model in one pass (prefill) and fill the cache.
2. Take the probabilities for the next token and sample one using settings such as temperature and top-p.
3. Append it, run one more step (decode) and repeat until a stop condition.

Step 10: Optional speed-ups

• Quantized cache: store the key-value cache in lower precision to fit longer contexts in the same memory.
• Speculative decoding: a small draft model proposes several tokens and the large model checks them in one pass. Accepted tokens save time.
• Dynamic batching and paged cache: share the cache across requests and reuse common prefixes.
• Constrained sampling: restrict output to match a grammar or a pattern.

A minimal Python sketch

The code below shows the general shape of loading a model and generating text. Names are shown for illustration. Check the library documentation of your installed version for exact class and argument names.

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2DynamicGenerator, ExLlamaV2Sampler

config = ExLlamaV2Config("/models/example-model-4.25bpw-exl2")
model = ExLlamaV2(config)
cache = ExLlamaV2Cache(model, max_seq_len=8192, lazy=True)
model.load_autosplit(cache)

tokenizer = ExLlamaV2Tokenizer(config)
generator = ExLlamaV2DynamicGenerator(model=model, cache=cache, tokenizer=tokenizer)

settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7
settings.top_p = 0.9

text = generator.generate(prompt="Explain quantization in one paragraph.",
                          max_new_tokens=200,
                          gen_settings=settings)
print(text)

A conversion command sketch

# measure and quantize to an average of 4.25 bits per weight
python convert.py \
  -i /models/example-model-fp16 \
  -o /work/exl2_tmp \
  -cf /models/example-model-4.25bpw-exl2 \
  -b 4.25

Command names and flags differ across versions. Treat this as a guide to the workflow, not exact syntax.

Key concepts and terms

Bits per weight in practice

These bands are rough guidance based on how low-bit methods generally behave. Test your own model and task, since larger models usually tolerate lower bit widths better than small ones.

A worked example

Suppose you want to run a model with about 30 billion parameters on a single consumer GPU with 24 GB of memory.

Step 1: Estimate the size

Memory for weights is roughly the number of parameters times the bits per weight divided by eight.

The arithmetic: 30 billion parameters at 4.5 bits is 30 times 4.5 divided by 8, about 16.9 GB. These are estimates. Real files add some overhead for scales and metadata.

Step 2: Leave room for the cache and activations

The key-value cache grows with context length and with the number of layers and heads. A long context can use several gigabytes. If weights take 17 GB on a 24 GB card, about 7 GB remain for the cache, activations and the system. That is enough for a moderate context. A quantized cache can stretch it further.

Step 3: Pick a target

A target near 4.5 bits per weight leaves a sensible margin. If you want a longer context, you might pick 4.0. If you want the best quality and have a bigger card, you might pick 5.0 or 6.0.

Step 4: Quantize

Run the converter with the target and a calibration dataset. The measurement pass takes the longest. It needs the full-precision model readable from disk and a GPU with enough memory to process layers one at a time.

Step 5: Evaluate

Compare the quantized model with the original on text that matters to you.

def perplexity(model, texts):
    total_nll, total_tokens = 0.0, 0
    for t in texts:
        nll, n = model.score(t)      # negative log-likelihood and token count
        total_nll += nll
        total_tokens += n
    return math.exp(total_nll / total_tokens)

orig = perplexity(fp16_model, eval_texts)
quant = perplexity(exl2_model, eval_texts)
print(f"original={orig:.3f} quantized={quant:.3f} gap={quant-orig:.3f}")

Perplexity is a quick check. It does not replace task tests. Run a handful of real prompts, check formatting, reasoning and long-context behaviour.

Step 6: Tune sampling and speed

Try a draft model for speculative decoding, a quantized cache and the maximum context you need. Measure tokens per second for a short prompt and for a long prompt, since prefill and decode behave differently.

What you learn

The example shows the main trade: lower bits fit bigger models and run faster per token, at some cost in fidelity. The mixed-precision format tries to get the best fidelity for each bit spent.

Strengths and limitations

Strengths

• Flexible bit widths. You can pick an average close to what your memory allows, not just a fixed 4-bit or 8-bit step.
• Sensitivity-aware allocation spends bits where they matter most.
• Fast single-stream generation on a single consumer GPU, thanks to tuned kernels.
• Supports quantized key-value cache for longer contexts.
• Supports speculative decoding and constrained output.
• Active use in local-model communities, with front ends and servers that embed it.
• Smaller files load quickly and cost less to store and share.

Limitations

• The ecosystem is centred on NVIDIA GPUs. Support on other hardware varies.
• It is not designed for large-scale multi-user serving, where engines with strong schedulers do better.
• Quantization is lossy. Low bit widths hurt small models more than large ones.
• Creating an EXL2 model needs a full-precision model, calibration data and time.
• Model architecture support depends on the version. Very new architectures may lag.
• File formats and APIs change between versions, so pinning versions matters.
• Fewer deployment tools than the largest serving engines, such as monitoring and autoscaling.
• Tool and format knowledge is spread across community documentation, so reading release notes is important.

When it is a good fit

How it compares with alternatives

ExLlamaV2 versus llama.cpp

Both aim to run local models. llama.cpp is built for portability: CPUs, many GPU types and unified memory machines, with the GGUF file format. ExLlamaV2 is built for speed on NVIDIA cards with a mixed-precision format. If you have one strong NVIDIA GPU and the whole model fits, ExLlamaV2 is often very fast. If you need CPU fallback or broad hardware, llama.cpp is more flexible.

ExLlamaV2 versus GPTQ

EXL2 builds on the same kind of error-compensating quantization as GPTQ, and ExLlamaV2 can also load GPTQ models. The main difference is flexibility: EXL2 mixes bit widths inside a model and lets you choose an average, while classic GPTQ uses a fixed width across the model.

ExLlamaV2 versus vLLM

vLLM schedules many requests at once with continuous batching and paged cache, and it targets throughput. ExLlamaV2 targets quick responses for one stream on small hardware. Choose by workload: one person chatting, or a service answering many people.

Weight-only versus weight-and-activation

ExLlamaV2 uses weight-only quantization: weights are stored in low bits, and activations stay in higher precision. This is simple and accurate. Methods such as SmoothQuant quantize activations too, which can speed up compute-bound work on hardware with fast low-precision units, at the cost of extra care with outliers.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing values and sometimes no useful column names. These models are usually pre-trained on large numbers of synthetic or real tables and then use in-context learning: they read labelled example rows in the prompt and predict for new rows without updating weights. TabPFN is a well-known example, and FT-Transformer is a related deep model for tabular inputs.

ExLlamaV2 is an inference tool for language models, so its link to tabular models is about deployment and about the language-model layer that often surrounds them. The connections below are concrete but modest, and none of them claim a built-in integration.

Why quantization matters for table models

Everything that makes ExLlamaV2 useful applies to any large transformer: weights dominate memory, and single-stream generation is limited by memory bandwidth. A table-oriented transformer that reads many rows as a long context also has a large key-value cache. The same ideas help here:

• Lower-bit weights reduce the memory needed for the model itself.
• A quantized cache lets more example rows fit in the context.
• Mixed precision protects the layers that handle numeric precision most.

Whether a specific tabular model can be run through this library depends on its architecture. ExLlamaV2 supports popular language-model families. Tabular models with different architectures need their own runtimes or general tools such as bitsandbytes and compilers such as TensorRT.

Language models around the table

Many data products pair a tabular model with a language model. The language model reads the user's question, writes a SQL query, chooses a prediction tool, and explains the result. Running that language model locally on a single GPU, with a quantized model served by ExLlamaV2, keeps sensitive data on the premises. That matters for row-level business data.

Serialising rows for a language model

When a language model must read a table directly, rows are turned into text. A compact serialisation uses fewer tokens, and a quantized cache helps hold more rows.

age=41 | plan=standard | usage=312 | tickets=2 | churned=NA
age=29 | plan=pro      | usage=815 | tickets=0 | churned=no

Quantization can slightly reduce numeric precision in a model's reasoning, so check results on number-heavy prompts. For exact arithmetic, call a calculator or SQL tool rather than relying on the model.

Missing values and absent column names

Language models can help infer what unnamed columns mean and propose imputation rules, which is related to semantic labelling work such as semantic sector heads. A local quantized model can do that labelling without sending data off site. The actual gap filling is better left to a dedicated step such as imputation networks or to the tabular model itself.

Anomaly detection with explanation

A detector, for example from the anomaly detection family, gives a score to each row. A local language model can then write a short explanation by reading the flagged row, its neighbours and any notes. The fast single-stream speed of a quantized model suits this interactive "explain this row" use.

Retrieval-augmented tabular workflows

Language models used in Naive RAG, Advanced RAG, Modular RAG and Agentic RAG can be served by a quantized local engine. In a tabular setting, such a model plans queries, picks context rows and explains predictions, while a tabular foundation model does the prediction.

Continual adaptation

Tabular data drifts. Because in-context learning uses rows from the prompt, fresh rows can be supplied at prediction time without retraining. A longer cache budget from quantization can allow more recent rows in context. This does not replace retraining when the data shape shifts a lot.

Honest limits

• ExLlamaV2 does not learn from tables and is not a tabular model.
• It runs language models. Support for other architectures should not be assumed.
• Quantization changes numeric behaviour a little. Verify on your own tasks.
• Row-level data is sensitive. Local inference helps, but access control still matters.

See /model/architecture, /model/research and /model/benchmarks for background on how table-focused models are built and measured.

Practical guidance and best practices

Choose the right bit width

• Estimate memory: parameters times bits divided by eight, plus the cache and overhead.
• Prefer a larger model at a moderate bit width over a smaller model at high precision when memory is fixed. Large models usually tolerate quantization better.
• Avoid going very low on small models. They lose quality fastest.
• Keep a margin of a few gigabytes for the cache, activations and the operating system.

Use good calibration data

• Use text that resembles your real prompts, such as chat, code or documents.
• Use enough variety to cover your use cases.
• Keep the calibration set separate from your evaluation set.
• If you serve a specific language or domain, include it.

Evaluate before trusting

Manage the cache

• Set the maximum sequence length to what you need, not the largest possible.
• Use a quantized cache when memory is tight, and check quality on long prompts.
• Reuse common prompt prefixes where the generator supports it.

Speed tuning

• Measure prefill speed and decode speed separately.
• Try speculative decoding with a small draft model from the same family and tokenizer.
• Close other GPU programs, since memory is shared.
• Keep drivers and the library version compatible.

Packaging and operations

• Pin library and driver versions, and record them with each model file.
• Keep the original model and quantization settings in a README beside the file.
• Version your quantized models and test after each change.
• For multi-user serving, put a proper server and scheduler in front, or choose a serving engine such as vLLM.

Security

• Treat model files like code. Download from trusted sources and check hashes.
• Keep local servers on a private network, with authentication if they must be reachable.
• When the model reads untrusted text, defend against injected instructions. See prompt injection defense.
• Add guardrails for sensitive applications.

Example configuration for a local service

service: local-llm
model_dir: /models/example-model-4.5bpw-exl2
max_seq_len: 8192
cache:
  type: quantized
  bits: 8
sampling:
  temperature: 0.7
  top_p: 0.9
  max_new_tokens: 512
draft_model:
  enabled: true
  dir: /models/example-draft-4.0bpw-exl2
server:
  host: example.internal
  port: 5000
  auth: required

Common mistakes

• Choosing a bit width without leaving room for the key-value cache, then running out of memory on long prompts.
• Quantizing a small model very low and blaming the tool for poor quality.
• Using calibration text that looks nothing like real prompts.
• Judging quality by perplexity alone.
• Expecting a single-stream engine to serve many users well.
• Mixing library versions and model files from different eras and getting load errors.
• Forgetting that the draft model for speculative decoding must share the tokenizer with the main model.
• Setting the context length far above need, which wastes memory.
• Comparing speed on short prompts only, then being surprised by slow long prompts.
• Trusting arithmetic from a quantized model without a calculator tool.
• Downloading model files from unknown sources.
• Exposing a local inference server to the open network without authentication.
• Assuming support for a brand new architecture without checking release notes.
• Treating quantization as free. It always trades some fidelity for size and speed.

Frequently asked questions

What is ExLlamaV2?

ExLlamaV2 is an inference library for running quantized large language models quickly on a single GPU. It comes with the EXL2 quantization format, which can use different bit widths for different parts of a model, and with GPU kernels tuned to read those packed weights.

What is EXL2?

EXL2 is the quantization format used by ExLlamaV2. It measures how sensitive each part of a model is to rounding and then assigns more bits to sensitive parts and fewer to tolerant parts, so the average bits per weight matches a target you choose.

How much memory do I need?

As a rule of thumb, weight memory is the number of parameters times the bits per weight divided by eight. Add room for the key-value cache, activations and system use. A model with 30 billion parameters at about 4.5 bits needs roughly 17 GB for weights, so a 24 GB card leaves a limited cache budget.

Is ExLlamaV2 faster than llama.cpp?

On a single NVIDIA GPU with a model that fits in memory, ExLlamaV2 is often very fast for one stream. llama.cpp is more portable and supports CPUs and many other devices. The best choice depends on your hardware and whether you need that portability.

Can ExLlamaV2 run on a CPU?

It is built around GPU kernels and is not meant for CPU-only machines. For CPU or mixed CPU and GPU use, a tool such as llama.cpp with GGUF files is the usual choice.

How does it differ from GPTQ?

Both use error-compensating post-training quantization. GPTQ usually applies one bit width across the model, while EXL2 mixes bit widths inside the model and lets you target an average. ExLlamaV2 can also load GPTQ models.

Does quantization hurt quality?

Yes, a little, and the loss grows as bit width falls. Larger models tolerate it better than small ones. Mixed precision reduces the loss by protecting sensitive layers. Always compare the quantized model with the original on your own tasks.

Can I fine-tune with ExLlamaV2?

No. It is an inference library. To fine-tune in low precision, look at methods such as QLoRA with a library like bitsandbytes, then quantize the merged model afterward if you want to run it with EXL2.

Is ExLlamaV2 good for serving many users?

It is better suited to one or a few users on one machine. For many concurrent requests, serving engines with continuous batching and paged cache management, such as vLLM, are designed for higher throughput.

Does it help with tabular foundation models?

Indirectly. It runs language models that often surround tabular models, for planning, labelling, query writing and explanation, and it keeps data local. It does not learn from tables, and support for non-language architectures should be checked before assuming it works.

Glossary

• Autoregressive generation: producing text one token at a time, each based on the earlier ones.
• Bits per weight: the average number of bits used to store each weight.
• Calibration data: sample text used to guide quantization.
• Decode: the stage that generates tokens one by one.
• Draft model: a small model that proposes tokens in speculative decoding.
• EXL2: the mixed-precision quantization format of ExLlamaV2.
• Group size: the number of weights that share one scale value.
• Key-value cache: stored attention keys and values from earlier tokens.
• Kernel: a small program that runs on the GPU.
• Memory bandwidth: how fast a GPU can read its memory.
• Mixed precision: using different bit widths for different parts of a model.
• Perplexity: a score of how well a model predicts text.
• Prefill: the stage that processes the prompt in one pass.
• Quantization: storing numbers with fewer bits.
• Speculative decoding: verifying draft tokens with a larger model.

Summary

ExLlamaV2 is an inference library and a quantization format, EXL2, built to run large language models quickly on a single GPU. It measures how sensitive each part of a model is, spends more bits where they matter, and uses tuned kernels to generate tokens with low memory use. The result is a flexible trade between size, speed and quality.

Use it when one person or a small group needs fast responses from a large model on a desktop-class NVIDIA card. Estimate memory with care, leave room for the cache, calibrate on realistic text and evaluate on real tasks. For high-concurrency serving or CPU use, look at other tools. For tabular work, treat it as a way to host the language model that plans, labels and explains, while a tabular foundation model does the table-level prediction. Related pages include GPTQ, AWQ, GGUF, vLLM and /model/architecture.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: