GGUF

Overview

GGUF is a binary file format for storing machine learning models, designed so that a model can be loaded quickly and run on ordinary hardware such as laptops, desktop CPUs and small edge devices. A single GGUF file holds the quantized weights, the tokenizer, the model settings and other metadata. You can copy one file, point a runtime at it, and start generating output, with no separate config files.

The format is best known as the native format of llama.cpp and tools built on it. Its value is practical. It keeps the whole package together, it supports many quantization levels that trade quality for size, and it is laid out so that the operating system can map the weights straight into memory. This page explains what is inside a GGUF file, how the quantization types work, how to create and use files, how GGUF compares with other formats, and how the same ideas matter for small models that work with tables.

What it is

GGUF stands for a general-purpose model file format that grew out of the GGML tensor library. It is a container. Like a zip file or a shipping box, it holds many things in a defined layout so that any program that knows the layout can open it.

A GGUF file contains:

• A header with a magic marker, a version number, the count of tensors and the count of metadata entries.
• A set of key-value metadata entries that describe the model: architecture name, number of layers, hidden size, number of attention heads, context length, normalization settings, rope settings, the tokenizer vocabulary, special token ids, and often a chat template.
• A table of tensor descriptions: for each tensor, its name, number of dimensions, shape, data type and offset into the data block.
• The tensor data itself, aligned to a fixed boundary so that it can be memory-mapped.

Why a single file matters

Before a unified container, model releases came as several files: weights in one or more shards, a config file, a tokenizer file, and special token files. If one was missing or mismatched, loading failed. With GGUF, the settings and the tokenizer travel with the weights. The file describes itself.

Quantization inside the file

A GGUF file can store weights in full precision, in 16-bit floats, or in one of many quantized types. Quantization replaces each group of weights with small integers and a scale, so the file is much smaller. A model that takes 14 GB in 16-bit might take around 4 GB at 4 bits per weight. Different tensors in one file can use different types. This lets a converter keep the most sensitive tensors at higher precision while compressing the rest.

What GGUF is not

It is not a training format. You do not train inside it. It is not a runtime: it needs a program that reads it. It is not a quantization algorithm by itself, although it defines the storage layouts for several algorithms. And it is not the only format for quantized models. Other methods such as GPTQ and AWQ have their own storage conventions, and general-purpose formats exist too.

Where it came from

The story starts with the GGML library, a tensor library in C written to run language models on CPUs with low overhead. It was created by the author of llama.cpp, and it quickly became popular for running open language models on consumer machines. GGML defined its own file format, and the format went through several revisions as features were added.

The early formats had problems. They stored settings in a fixed order, so adding a new setting could break old files. They needed hard-coded knowledge of each model. Users ended up with files that looked similar but did not load in newer programs.

In 2023, the project introduced GGUF as a replacement. The goals were extensibility, a clean separation of metadata from tensor data, and better support for many architectures through named key-value pairs. New metadata keys can be added without breaking old readers, because unknown keys can be skipped. The format also made room for storing the tokenizer in the file.

Since then, GGUF has spread beyond the original project. Many desktop apps and local model runners support it. Model hubs host large numbers of GGUF files, usually produced by community members who convert and quantize popular models. Support also appeared in libraries such as Transformers, which can read GGUF files and turn them back into regular model objects for some architectures.

Details of specific quantization types have been refined over time. Newer types, such as the K-quant family and importance-weighted types, improved the quality at a given size. Always check the project documentation for the exact list that your version supports.

How it works

To see how GGUF works, imagine moving house with a moving company. The company wants a clear label on every box, an inventory list on the truck and a layout where boxes can be unloaded straight into the right rooms. GGUF is the inventory list, the labels and the packing plan for a model.

Step 1: read the header

The reader opens the file and checks the magic bytes, which identify the format. It then reads the version. Versions let the format evolve. The header also holds how many tensors and how many metadata entries to expect.

Step 2: read the metadata

Metadata entries are key-value pairs. Keys are strings with dotted names, grouped by purpose. For example, general keys describe the architecture and name, architecture-specific keys describe layer counts and sizes, and tokenizer keys hold the vocabulary and merge rules. Values can be integers, floats, booleans, strings or arrays of these.

The runtime looks up the architecture key and then reads the matching settings. That is how one program supports many model families: the file tells it which architecture to build and with what sizes.

Step 3: read the tensor table

The tensor table lists every tensor with a name such as a layer's attention projection, its shape, its data type and its byte offset. The reader does not load the data yet. It only learns where everything is.

Step 4: map the data

The tensor data block follows, aligned to a boundary. Because the layout is known and aligned, the runtime can ask the operating system to map the file into memory. Pages are loaded only when touched. This has several benefits:

• Startup is fast because nothing is copied up front.
• Several processes can share the same mapped weights.
• If memory is tight, the system can evict and reload pages from disk.

Step 5: build the model graph

With sizes and tensor locations known, the runtime builds the compute graph for the architecture. Each operation, such as a matrix multiplication, is given pointers to the mapped tensor data and runs a kernel that understands the tensor's type.

Step 6: run kernels on quantized data

Quantized tensors are not turned back into 16-bit floats in full. Instead, kernels read blocks of quantized integers and their scales and do the multiply-add directly, converting on the fly. On CPUs, kernels use vector instructions. On GPUs and other accelerators, backends provide their own kernels. Some layers can be split between CPU and GPU, so a model larger than GPU memory can run with part of it offloaded.

How block quantization works

Most GGUF quantization types use blocks. A tensor is cut into small groups of values, such as 32 values per block, or into super-blocks made of several smaller blocks. For each block, the format stores a scale and the quantized integers. To recover a value, multiply the integer by the scale, and add an offset in some types.

An everyday analogy is a map with a scale bar. Instead of writing every distance in exact kilometers, you write small whole numbers of map units and keep one scale for the whole map. A block is a small map with its own scale bar, so a region with large distances and one with small distances can each use their own scale and stay accurate.

The K-quant idea

K-quant types use two levels. Values are grouped in small blocks with their own scale, and those blocks are grouped in a super-block whose scales are themselves quantized. This lowers the overhead of storing scales and allows mixed precision inside a model: for example, a medium K-quant type may give some tensors a higher bit width than others, since a few tensors are known to be more sensitive to error.

Importance-weighted types

Some newer types use an importance matrix. A converter runs sample text through the model, measures how much each weight matters to the output, and picks quantization values that preserve the important weights better. This is conceptually close to the calibration used by GPTQ and AWQ, and it is especially helpful at very low bit widths.

Conversion workflow

To create a GGUF file you usually start from a model in a standard format.

1. Convert the original weights, tokenizer and config into a GGUF file in 16-bit precision.
2. Optionally compute an importance matrix from sample text.
3. Run a quantization tool that rewrites the file with the chosen type.
4. Test the result on sample prompts and a perplexity or task measure.

Key concepts and terms

A worked example

This example converts a model, quantizes it, runs it, inspects it and does a hand calculation of file size. Paths and names are placeholders, and command names may differ slightly by version.

Hand calculation of size

Suppose a model has 7 billion weights.

• At 16 bits per weight, the size is 7 billion times 2 bytes, about 14 GB.
• At 8 bits per weight, about 7 GB.
• At about 4.5 bits per weight, which is typical for a medium 4-bit K-quant once scales are counted, about 7 billion times 4.5 divided by 8, which is around 3.9 GB.
• At about 3 bits per weight, about 2.6 GB.

These are estimates. The real file also stores the tokenizer and some tensors at higher precision, so sizes are a little larger. Notice that the effective bits per weight is higher than the nominal number, because scales take space.

Memory use at run time is the file size plus the cache of past keys and values, which grows with context length, plus working buffers.

Converting and quantizing

# Convert from a standard checkpoint folder to a 16-bit GGUF file.
python convert_hf_to_gguf.py /models/example-model \
  --outfile /models/example-model-f16.gguf --outtype f16

# Optional: build an importance matrix from sample text.
./llama-imatrix -m /models/example-model-f16.gguf \
  -f /data/calibration.txt -o /models/imatrix.dat

# Quantize to a medium 4-bit K-quant type.
./llama-quantize --imatrix /models/imatrix.dat \
  /models/example-model-f16.gguf \
  /models/example-model-Q4_K_M.gguf Q4_K_M

Running it

# Interactive run on CPU, with some layers offloaded to a GPU if present.
./llama-cli -m /models/example-model-Q4_K_M.gguf \
  -c 4096 -ngl 20 -p "Explain what a tabular model is."

# Start a local server with an HTTP API.
./llama-server -m /models/example-model-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 -c 4096

Calling the local server

import json
import urllib.request

req = urllib.request.Request(
    "http://127.0.0.1:8080/v1/chat/completions",
    data=json.dumps({
        "messages": [{"role": "user", "content": "Summarize this column: ages 20-60"}],
        "max_tokens": 128,
    }).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as r:
    print(json.loads(r.read())["choices"][0]["message"]["content"])

Inspecting a file with Python

from gguf import GGUFReader

reader = GGUFReader("/models/example-model-Q4_K_M.gguf")
for key in list(reader.fields)[:15]:
    print(key)
for t in reader.tensors[:5]:
    print(t.name, t.shape, t.tensor_type)

Inspection is the fastest way to answer questions such as which architecture a file uses, what the context length is, and which tensors are stored at which precision.

Comparing quantization levels

Make a small table for your own use case.

Fill in the middle column with a measurement on your own data. Do not rely on generic claims.

Layout of a file, drawn as text

+---------------------------+
| magic bytes, version      |
| tensor count, kv count    |
+---------------------------+
| metadata key-value pairs  |
|   general.architecture    |
|   <arch>.block_count      |
|   <arch>.context_length   |
|   tokenizer.ggml.tokens   |
|   ...                     |
+---------------------------+
| tensor table              |
|   name, dims, type, offset|
+---------------------------+
| padding to alignment      |
+---------------------------+
| tensor data blocks        |
+---------------------------+

Reading top to bottom, a program learns what the model is, how big it is, where each tensor lives and then maps the data. The key names shown here are examples of the naming style, and exact keys depend on the architecture and the format version.

Everyday comparison: an e-book versus a printed pile

A model with separate weight, config and tokenizer files is like a book delivered as loose chapters, a table of contents on a separate sheet and a glossary in another envelope. GGUF is the bound book with its own contents page. Quantization is like printing the book in a smaller, more compact edition: it is lighter to carry, and a few fine details in the pictures are lost.

Choosing a type for a table-related language task

For tasks such as column typing, sector classification and short summaries, a medium 4-bit or 5-bit type is usually enough, since the answers are short and the task is forgiving. For tasks that generate code or queries over tables, test a higher level, because small errors in syntax can break the output. For tasks that return numbers, validate every number against the data, whatever the quantization level.

Testing plan for a new file

1. Load the file and print the metadata. Confirm the architecture, context length and tokenizer.
2. Run five prompts that you know the answer to, and read the outputs.
3. Run a small batch of your task and compute accuracy.
4. Measure memory and tokens per second.
5. Compare with the next higher quantization level. If there is no visible gain, keep the smaller file.
6. Record the results with the file hash.

Strengths and limitations

Strengths

• One file holds everything needed to run the model.
• Fast startup through memory mapping.
• Wide hardware reach: CPUs with vector instructions, many GPUs, and mobile or edge devices through the runtime backends.
• Many quantization levels, from near-lossless to very small.
• Extensible metadata. New keys can be added without breaking old readers.
• A large ecosystem of tools, apps and ready-made files.
• Mixed precision inside one file, so sensitive tensors can stay at higher precision.
• Friendly to partial offload, which lets a model larger than GPU memory still run.

Limitations

• Mainly aimed at inference. Training and full fine-tuning are not done in the format, although conversion tools exist for adapters.
• Architecture support depends on the runtime. A new model family needs code in the runtime before a GGUF file for it will run.
• Quantized files lose some accuracy, and the loss is larger at very low bit widths and for small models.
• Speed on a server GPU may trail dedicated engines such as vLLM when serving many users, because those engines focus on batching.
• File names carry type labels by convention only. Always check the metadata rather than trusting a name.
• Different versions of the runtime may handle edge cases differently, such as chat templates and special tokens.
• Not every quantization method from other ecosystems has a GGUF equivalent.

How it compares with alternatives

A simple rule helps. If you want to train or fine-tune, work in a standard framework format, with LoRA or QLoRA as needed, then convert to GGUF at the end for local use. If you want to serve many users on server GPUs, look at dedicated serving engines first.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories, dates and text, missing values, and sometimes columns without names. A model like TabPFN reads labeled rows as context and predicts new rows without updating weights. GGUF is not a table format and does not define a model. It is a way to package and run models compactly. Its role is on the deployment side, and it touches tabular work in several concrete ways.

Running language models next to tables

Many table workflows use a language model for the text-heavy parts: reading column names and descriptions, guessing the semantic type of a column, mapping a table to an industry sector, writing a plain-language summary of findings, or turning a question into a query. A quantized language model in GGUF form can run locally, next to the data, on a modest machine. This keeps sensitive tables in place, since no table leaves the computer. It also removes network latency for many small calls.

Semantic and sector classification at the edge

Classifying a column as an email, a date, a currency or an identifier, or classifying a whole table by sector, needs only a small to medium language model. At 4 or 5 bits per weight, such a model fits in a few gigabytes. Batch jobs can then classify thousands of columns on a CPU server without a GPU.

Packaging small table models

Some table models are small enough to run on a CPU. A runtime built on the same tensor library can, in principle, host any model architecture for which operators and graph code exist. Whether a given tabular architecture has GGUF support depends on the runtime and the model family, so do not assume it. Where support is absent, other deployment formats such as ONNX Runtime may be the better route.

Quantization and numeric tasks

Tables are numeric. Quantizing a language model is usually safe for text tasks, but quantizing a model that predicts numbers needs more care. Small shifts in weights can move predicted values or probabilities. If a table model is quantized, evaluate calibration, error on regression targets, and agreement of class predictions with the full-precision model on held-out tables. Keep sensitive layers, such as embedding and output heads for numeric values, at higher precision using the mixed-type storage that GGUF allows.

Missing values, imputation and anomaly detection

A local model can propose values for missing cells, flag cells that look wrong, or explain why a row looks unusual. When the model is quantized, you should test these tasks the same way as the full model: mask known cells, ask the model to fill them, and compare accuracy at each quantization level. The loss from quantization is often small, but it is not zero, and it should be measured on your own tables.

Continual learning and adapters

Fine-tuning happens in a standard framework, often with adapters such as LoRA. After training, adapters can be merged into the base weights and the result converted to GGUF, or applied as separate adapter files at load time by runtimes that support it. This supports a cycle: learn from new tables in the training stack, package the update as a new file, deploy it to local machines, and keep earlier versions for rollback.

Privacy and governance

Because a GGUF model runs offline, it fits workflows where table data cannot be sent to external services. File hashes and metadata help with governance: you can record which file produced which output. Keep a registry of file name, hash, base model, quantization type and evaluation results. For model-side context, see /model/research, /model/architecture and /model/benchmarks.

Practical guidance and best practices

Choose the quantization level by measurement

• Start with a 5-bit or 4-bit medium K-quant. These are widely used and balanced.
• Move up to 6-bit or 8-bit if quality on your task drops.
• Move down to 3-bit or lower only if you must, and test carefully, especially with small models.

Match the file to your memory

Add the file size, the cache for the context you plan to use and some overhead. If the sum is close to your memory limit, reduce context length or choose a smaller type.

Use partial offload when needed

If a GPU is available but too small, offload as many layers as fit. Speed grows with the number of offloaded layers, so tune this number on your hardware.

Pay attention to context length

The cache of past keys and values grows with context length and can rival the weights in size at long contexts. Set the context length to what you need, not the maximum the model allows. The cache can itself be stored in lower precision in some runtimes.

Use the right chat template

Instruct and chat models expect a specific prompt format. The template is stored in the metadata of many files. If outputs look odd, check that the runtime applies the template, or that you pass your own correctly.

Keep the full-precision source

Store the original weights safely. You can regenerate GGUF files when quantization methods improve or a bug is fixed. Treat GGUF files as build artifacts.

Verify files

Record a checksum for each file you download or produce. Prefer files from sources you trust. A GGUF file is data, not code, but a damaged file can still fail in confusing ways, and a model from an unknown source can contain unwanted behavior.

Evaluate on your task

Run a small test set before and after quantization. For language tasks, compare task accuracy and a perplexity number. For table tasks, compare predictions on held-out rows and check calibration.

Keep runtime and format versions in sync

Newer files may need newer runtimes. When something fails to load, check the format version in the header and update the runtime.

Consider speed settings

Set the number of threads to match your physical cores, not hyper-threads, as a starting point. Use batch settings suited to prompt length. Measure tokens per second for prompt processing and for generation separately, because they behave differently.

Document your pipeline

Write down the source model, the conversion command, the importance data used, the quantization type, the runtime version and the evaluation results. This makes it easy to rebuild or audit a file later.

Estimating run-time memory

Memory at run time has three parts: the weights, the key-value cache and working buffers.

1. Weights: about the file size, though memory mapping may load them lazily.
2. Cache: for each layer, two tensors (keys and values) per token. The size is roughly 2 times layers times context length times key-value width times bytes per value. Models with grouped-query attention have a smaller key-value width, so their cache is much smaller.
3. Buffers: scratch space for activations, usually small compared with the other two.

Take a model with 32 layers, a key-value width of 1,024 and a context of 8,192 tokens, with the cache stored in 16-bit values. The cache is 2 times 32 times 8,192 times 1,024 times 2 bytes, which is about 1.07 GB. Add that to a 4 GB weight file and you need around 5 GB plus buffers. Doubling the context doubles the cache, so long contexts need planning. The numbers are an illustration of the method.

Reading the type codes

Besides the K-quant names, you may see simple types. The tensor table stores a code for each tensor.

A single file often mixes these: norms in F32, most matrices in a K-quant type, and the output matrix in a higher precision type.

Tokenizer data inside the file

The vocabulary, token scores, merge rules and special token ids are stored as metadata arrays. This is why you can run a file without a separate tokenizer file. A mismatch between the tokenizer in the file and the one used at training time is a common cause of odd outputs, so when you convert a model, make sure the converter uses the correct tokenizer files and compare token counts on a sample string.

Shards and splitting

Very large models can be split into several GGUF files, each holding some tensors, with metadata that says how many parts there are. Runtimes load the parts together. Splitting helps with download limits and file system limits. Keep all parts together and check that the part count in the names matches.

LoRA adapters in GGUF form

Adapters can be converted into their own GGUF files. A runtime can apply the adapter on top of a base file at load time, with an optional scale. This avoids storing a full copy of the model for each task. For table-related uses, you could keep one base language model and several small adapters, for example one for column typing and one for sector classification, and switch between them.

Operational checklist for local deployment

• Pin the runtime version and the file hash.
• Bind servers to the local interface unless you add authentication and encryption.
• Set limits on context length and output length.
• Log prompt and output sizes and timing, but avoid logging sensitive table content.
• Test a restart: the model should reload quickly thanks to memory mapping.
• Keep a rollback file in case a new quantization behaves worse.
• Monitor memory, since several models loaded at once can compete for it.

Common mistakes

• Assuming a smaller file is always better. Quality drops at low bit widths, especially for small models.
• Trusting the file name instead of the metadata for type and architecture.
• Running out of memory because the cache for a long context was not counted.
• Using a base model file as if it were a chat model, or the reverse, and blaming the format for poor replies.
• Ignoring the chat template and getting garbled or off-topic output.
• Quantizing from an already quantized file and compounding the error. Always start from the highest precision source you have.
• Comparing speed numbers across machines or settings that differ in threads, batch size or offload.
• Expecting server-grade throughput for many parallel users from a desktop-oriented runtime without testing.
• Forgetting to update the runtime when a new file version appears.
• Evaluating only with a generic benchmark and skipping the real task.
• Treating quantization as safe for numeric prediction without measuring calibration and error.
• Deleting the original weights and then being unable to rebuild when needed.

Frequently asked questions

What does GGUF stand for and what is it?

GGUF is a binary file format for packaging a machine learning model into one file, with its weights, tokenizer and settings together. It was introduced for the llama.cpp project as a successor to older GGML formats, and it is built so that the file can be loaded quickly and read by many programs.

What is the difference between GGUF and GGML?

GGML is the tensor library and the name of the older file formats that grew around it. GGUF is the newer file format that replaced them. GGUF stores settings as named key-value pairs, which lets new fields be added without breaking old readers, while the older formats used a fixed layout that was easy to break.

What do names like Q4_K_M mean?

The name describes the quantization type. The number is roughly the bits per weight, the letter K marks the family that uses two-level block scales, and the last letter marks a size variant, such as small, medium or large, which sets how many sensitive tensors keep higher precision. Larger variants are slightly bigger and usually a little more accurate.

Which quantization level should I choose?

For most people, a medium 4-bit or 5-bit type is a good starting point because it cuts size by a large factor with a small quality loss. Choose a higher level if your task is sensitive or your model is small, and a lower level only when memory forces you. Always test on your own task rather than relying on general claims.

Can I train or fine-tune a model in GGUF?

GGUF is built for inference, so the usual path is to train or fine-tune in a standard framework and then convert the result. Parameter-efficient methods such as LoRA are convenient here, because the small adapters can be merged into the base model or applied at load time before running in GGUF form.

Does GGUF work on GPUs?

Yes. The runtimes that read GGUF have backends for several kinds of GPUs, and you can offload some or all layers to the GPU. The format itself is hardware neutral, since it only stores data and metadata. Speed and support depend on the backend and your device.

How is GGUF different from safetensors?

Safetensors is a safe, fast format for storing tensors, mostly in full or half precision, and it usually leaves the tokenizer and settings in separate files. GGUF bundles tensors, tokenizer and settings in one file and defines block quantization types that runtimes can compute on directly. They serve different stages: safetensors for training and sharing, GGUF for compact local inference.

Is a GGUF model less accurate than the original?

Quantized models lose some accuracy because weights are stored with fewer bits. At 8 bits the loss is usually tiny, at 4 to 5 bits it is often small, and at lower bit widths it grows. The effect depends on the model size, the quantization method and the task, so measure it on your data.

Can GGUF help with tabular data work?

It helps indirectly. A quantized language model in GGUF form can run locally to classify column meanings, assign sectors, summarize results or suggest values for missing cells, which keeps data on the machine. For models that read tables directly, support depends on whether the runtime implements the architecture, so check before planning around it.

Is it safe to download GGUF files from the internet?

A GGUF file stores data and not executable code, which makes it safer than formats that can run code on load. Even so, you should use trusted sources, check hashes, keep the runtime up to date, and test model behavior, since a file can still contain a poor or altered model.

Glossary

• Block: a small group of weights that share one scale.
• Chat template: a stored recipe for turning a conversation into the text the model expects.
• Context length: the number of tokens the model can attend to at once.
• GGML: the tensor library that GGUF came from.
• GGUF: a single-file container for model weights, tokenizer and settings.
• Importance matrix: data that tells the quantizer which weights matter most.
• K-quant: a family of quantization types with two-level scales.
• Key-value cache: stored keys and values of past tokens that speed up generation.
• llama.cpp: a popular runtime that reads and runs GGUF files.
• Memory mapping: letting the operating system load parts of a file only when needed.
• Metadata: named settings stored inside the file.
• Offload: running some layers on a GPU while the rest run on the CPU.
• Perplexity: a language model score used to track quality loss, where lower is better.
• Quantization: storing numbers with fewer bits.
• Scale: a multiplier used to turn stored integers back into approximate real values.
• Tensor table: the list of tensors in the file with their shapes, types and offsets.

Summary

GGUF is a single-file, extensible container for quantized model weights, tokenizer and settings, created for llama.cpp and now supported across a large local-inference ecosystem. It is fast to load, runs on CPUs and many GPUs, and offers many quantization levels that trade size for quality. It is aimed at inference, so training happens elsewhere and the result is converted at the end.

For tabular foundation models, GGUF matters on the deployment side. It lets compact language models run beside the data for column typing, sector classification, summaries and imputation hints, it keeps sensitive tables local, and it fits an adapter-based continual learning cycle where updates are merged and repackaged. Numeric models need careful checks of calibration after quantization, and support for table-specific architectures depends on the runtime. Related pages include llama.cpp, GPTQ, AWQ, bitsandbytes, QLoRA and TabPFN, and model-side background is at /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to Quantization in my stack, under LLMs, retrieval, and evaluation. Others in the same group: