llama.cpp

Overview

llama.cpp is an open-source engine, written in C and C++, for running large language models on ordinary hardware. It loads models stored in the GGUF file format, usually quantised to 4, 5, 6 or 8 bits per weight, and generates text on CPUs, Apple Silicon, NVIDIA and AMD GPUs and other devices. It needs no Python and few dependencies, which makes it practical on laptops, desktops, small servers, phones and single-board computers.

An everyday picture: a portable camping stove. A restaurant kitchen is faster and can feed hundreds, but it needs a building, gas lines and staff. A camping stove is small, runs almost anywhere and cooks a good meal for a few people. llama.cpp is the camping stove of language models. A data-centre engine such as vLLM serves thousands of users on large GPUs; llama.cpp lets one person or a small team run a capable model on the machine in front of them, even without internet access.

Many popular local AI applications, including Ollama, LM Studio and others, use llama.cpp or its underlying tensor library, ggml, as their engine. That makes it one of the most important pieces of the local and offline AI ecosystem.

What it is

llama.cpp is released under the MIT licence.

Where it came from

Georgi Gerganov released llama.cpp in March 2023, shortly after Meta's LLaMA model weights became available to researchers. He had earlier written whisper.cpp, a C++ port of the Whisper speech model, using his ggml library. The first version showed that a 7-billion-parameter model could run at usable speed on a MacBook CPU with 4-bit quantisation, which surprised many people at the time.

A large community formed quickly, adding GPU back-ends, many model architectures and better quantisation schemes. In August 2023 the project introduced GGUF, replacing the earlier GGML file format with a more extensible single-file design that stores metadata such as the tokenizer and chat template. Gerganov founded ggml.ai to support development. In February 2026 the ggml.ai team joined Hugging Face, with the projects remaining open source.

How it works

Step 1: get a GGUF model

Many models are published directly as GGUF files on Hugging Face, often in several quantisation levels. Otherwise, a conversion script turns a Hugging Face checkpoint into GGUF at 16-bit precision, and llama-quantize reduces it to the chosen type.

Step 2: choose a quantisation

Quantisation stores weights in fewer bits. A 7-billion-parameter model needs about 14 GB at 16 bits but roughly 4 to 5 GB at about 4.5 bits per weight. K-quants group weights into blocks with shared scales, and mixed types keep more sensitive tensors at higher precision. Lower bits mean smaller files and faster generation, with some loss of quality that grows as bits fall.

Step 3: load and offload

When the model loads, llama.cpp maps the file into memory. On a machine with a GPU, a chosen number of layers can be offloaded to the GPU with the -ngl option, while the rest run on the CPU. On Apple Silicon, unified memory lets the GPU use most of system memory.

Step 4: generate

The engine processes the prompt, stores the KV cache, then generates tokens one at a time. Speed on a CPU is mainly limited by memory bandwidth, because each token requires reading all active weights. That is why quantisation helps so much.

Step 5: serve

llama-server exposes an OpenAI-compatible API, so the same client code used for hosted models can call a local one.

llama-server -m ./models/model-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080

This starts a server with an 8,192-token context and as many layers on the GPU as fit.

Key concepts and terms

A worked example

This scenario is illustrative. A small accounting firm wants a private assistant to summarise client emails without sending data to the cloud.

1. They choose an open 8-billion-parameter instruction model and download its Q4_K_M GGUF file, about 5 GB.
2. On an office desktop with 32 GB of memory and a mid-range GPU with 8 GB, they run llama-server with all layers offloaded. Generation runs at roughly 40 to 60 tokens per second; on the CPU alone it is closer to 8 to 12.
3. Their email tool already speaks the OpenAI API, so they point it at the local server's address.
4. Testing 50 real emails, Q4_K_M summaries are rated almost the same as Q8_0, while using about 40 per cent less memory, so they keep Q4_K_M.
5. A JSON grammar makes the assistant return a fixed structure of sender, topic and action items, which their scripts parse reliably.
6. Nothing leaves the building, which satisfies their client confidentiality policy.

Strengths and limitations

Strengths:

• Runs capable models on laptops, desktops and small servers, fully offline.
• Very broad hardware support, including CPUs and Apple Silicon.
• Small, dependency-light C and C++ code base.
• Many quantisation options to fit models into limited memory.
• OpenAI-compatible server and a huge model ecosystem in GGUF.

Limitations:

• Not designed for very high multi-user throughput like data-centre engines.
• Aggressive quantisation can noticeably reduce quality.
• Support for brand-new architectures may lag behind their release.
• Many options and frequent changes can confuse new users.
• CPU-only speed is modest for large models.

How it compares with alternatives

Practical guidance and best practices

1. Start with a Q4_K_M or Q5_K_M quantisation, then compare against Q8_0 on your own tasks.
2. Offload as many layers to the GPU as memory allows.
3. Set the context size to what you need; larger contexts use more memory for the KV cache.
4. Use the model's own chat template, which GGUF files usually include.
5. Use grammars or JSON schemas for any output a program must parse.
6. Download models from trusted publishers and check file hashes.
7. Keep llama.cpp updated for new models and fixes, but pin a version for production tools.
8. Bind the server to localhost or protect it with an API key when on a network.

Common mistakes

• Choosing the smallest quantisation without testing quality.
• Setting a huge context size and running out of memory.
• Forgetting -ngl, so a capable GPU sits unused.
• Using the wrong chat template, which produces odd or repetitive answers.
• Exposing the server on a public network without protection.
• Expecting data-centre throughput from a single desktop.

Frequently asked questions

Do I need a GPU to use llama.cpp?

No. It runs on CPUs alone, and is especially effective on Apple Silicon. A GPU makes generation much faster, and layers can be split between CPU and GPU.

What is GGUF?

GGUF is the single-file format llama.cpp uses for models. It stores the weights, often quantised, together with the tokenizer, chat template and other metadata needed to run the model.

How much memory do I need?

Roughly the size of the GGUF file plus room for the KV cache and the operating system. A 7 to 8 billion parameter model at 4-bit quantisation typically needs about 5 to 6 GB, more for long contexts.

Is llama.cpp only for Llama models?

No. Despite the name, it supports many model families, including Mistral, Qwen, Gemma, Phi and mixture-of-experts models, as well as some vision-language models.

How is llama.cpp related to Ollama?

Ollama is a separate application that makes downloading and running local models easy. It has used llama.cpp and ggml as its inference engine, adding its own model management and API.

Glossary

• Local inference: running a model on your own device rather than a remote service.
• SIMD: CPU instructions that process several numbers at once.
• Metal: Apple's GPU programming interface.
• Vulkan: a cross-vendor graphics and compute interface used as a back-end.
• Unified memory: memory shared by CPU and GPU, as on Apple Silicon.
• Tokens per second: a measure of generation speed.
• Perplexity: a measure often used to compare quantisation quality.

Summary

llama.cpp is a lightweight C and C++ engine that runs quantised language models in the GGUF format on CPUs, Apple Silicon and many GPUs, making private, offline AI practical on everyday machines. Released by Georgi Gerganov in March 2023, it introduced GGUF later that year, powers much of the local AI ecosystem, and has been supported by Hugging Face since early 2026. It is the right tool for local and small-scale use; choose quantisation by testing quality, offload layers to the GPU where possible, and protect the server if it is reachable on a network.

Related

It belongs to Inference and serving in my stack, under MLOps and AI infrastructure. Others in the same group: