Glossary · MLOps and AI infrastructure
What is vLLM? Definition and meaning
An engine that uses PagedAttention and continuous batching to serve language models efficiently.
Definition
vLLM: An engine that uses PagedAttention and continuous batching to serve language models efficiently.
In short: High-throughput LLM inference engine.
When it is used
Serving large language models in production.
Where it fits
vLLM belongs to Inference and serving in the MLOps and AI infrastructure topic. These are the other terms in the same group:
- Triton: A server that runs models from many frameworks with dynamic batching and concurrent execution on GPUs.
- SGLang: A serving framework with efficient prefix caching and a language for structured generation.
- TensorRT: A compiler and runtime that optimizes neural networks for NVIDIA GPUs.
- ONNX Runtime: A runtime that executes models in the ONNX format across hardware backends.
- llama.cpp: A library that runs quantized language models on CPUs and a range of GPUs.
Learn more
The full guide to vLLM explains how it works, when to use it and how it compares with the alternatives, with examples. The glossary lists every term on the site from A to Z.