# What is llama.cpp? Definition and meaning

A library that runs quantized language models on CPUs and a range of GPUs.

## Definition

**llama.cpp**: A library that runs quantized language models on CPUs and a range of GPUs.

In short: Efficient LLM inference in C++.

## When it is used

Running models locally and on the edge.

## Where it fits

llama.cpp belongs to **Inference and serving** in the [MLOps and AI infrastructure](/topics/mlops-ai-infrastructure) topic. These are the other terms in the same group:

- **[Triton](/glossary/triton)**: A server that runs models from many frameworks with dynamic batching and concurrent execution on GPUs.
- **[vLLM](/glossary/vllm)**: An engine that uses PagedAttention and continuous batching to serve language models efficiently.
- **[SGLang](/glossary/sglang)**: A serving framework with efficient prefix caching and a language for structured generation.
- **[TensorRT](/glossary/tensorrt)**: A compiler and runtime that optimizes neural networks for NVIDIA GPUs.
- **[ONNX Runtime](/glossary/onnx-runtime)**: A runtime that executes models in the ONNX format across hardware backends.

## Learn more

The full guide to **[llama.cpp](/tools/llama-cpp)** explains how it works, when to use it and how it compares with the alternatives, with examples. The [glossary](/glossary) lists every term on the site from A to Z.

Previous term: [LLaMA-Factory](/glossary/llama-factory) · Next term: [LlamaIndex](/glossary/llamaindex)
