Özer Cem KELAHMETKELAHMET
Glossary · MLOps and AI infrastructure

What is llama.cpp? Definition and meaning

A library that runs quantized language models on CPUs and a range of GPUs.

Definition

llama.cpp: A library that runs quantized language models on CPUs and a range of GPUs.

In short: Efficient LLM inference in C++.

When it is used

Running models locally and on the edge.

Where it fits

llama.cpp belongs to Inference and serving in the MLOps and AI infrastructure topic. These are the other terms in the same group:

  • Triton: A server that runs models from many frameworks with dynamic batching and concurrent execution on GPUs.
  • vLLM: An engine that uses PagedAttention and continuous batching to serve language models efficiently.
  • SGLang: A serving framework with efficient prefix caching and a language for structured generation.
  • TensorRT: A compiler and runtime that optimizes neural networks for NVIDIA GPUs.
  • ONNX Runtime: A runtime that executes models in the ONNX format across hardware backends.

Learn more

The full guide to llama.cpp explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.

Previous term: LLaMA-Factory · Next term: LlamaIndex

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open