# vLLM

vLLM is an open-source engine for fast, memory-efficient LLM inference using PagedAttention and continuous batching, with an OpenAI-compatible server.

## Overview

vLLM is an open-source engine for running large language models quickly and efficiently. It takes a model's weights, usually from [Hugging Face](/tools/hugging-face), loads them onto GPUs or other accelerators, and generates text for many users at once. It can be used as a Python library for batch jobs or as a server with an OpenAI-compatible API, so applications written for hosted APIs can switch to self-hosted models with little change.

Its reputation rests on two ideas. PagedAttention manages the memory that holds each conversation's intermediate state, the [KV cache](/tools/kv-cache), in small fixed-size blocks rather than large reserved chunks, which wastes far less memory. Continuous batching adds and removes requests from the running batch at every generation step, so the GPU never sits idle waiting for the slowest request to finish.

An everyday picture: a car park. Old systems reserved a whole row for each car in case it brought a trailer, so most spaces stayed empty. vLLM assigns individual spaces as they are needed, anywhere in the car park, and keeps a ticket that records where each part is parked. Far more cars fit in the same area, and new cars can enter as soon as others leave.

## What it is

| Part | Purpose |
|---|---|
| Engine | Schedules requests, runs the model and manages memory |
| PagedAttention | Stores the KV cache in fixed-size blocks mapped through a block table |
| Continuous batching | Adds new requests to the running batch at every step |
| Prefix caching | Reuses cached blocks for prompts that share the same beginning |
| OpenAI-compatible server | `vllm serve` exposes chat, completions and embeddings endpoints |
| Offline API | The `LLM` class for batch generation in Python |
| Parallelism | Tensor, pipeline, data and expert parallelism across GPUs and nodes |
| Quantisation support | Runs models quantised with methods such as [GPTQ](/tools/gptq), [AWQ](/tools/awq) and FP8 |
| Speculative decoding | Uses draft tokens to speed up generation |
| Structured outputs | Constrains output to JSON schemas, regular expressions or grammars |
| Multi-LoRA | Serves many LoRA adapters on one base model |

vLLM supports a wide range of model architectures, including dense transformers, [mixture-of-experts](/tools/mixture-of-experts) models, multimodal models and embedding models, and runs on NVIDIA and AMD GPUs and other accelerators through hardware plug-ins.

## Where it came from

vLLM was created at the University of California, Berkeley, in the Sky Computing Lab, by Woosuk Kwon, Zhuohan Li and colleagues working with Ion Stoica and others. The code was released in June 2023, and the paper "Efficient Memory Management for Large Language Model Serving with PagedAttention" was presented at the SOSP conference in 2023. It showed large throughput gains over earlier serving systems by reducing memory waste in the KV cache.

The project grew quickly into a broad community effort, with contributions from many companies and hardware vendors. In 2025 it became a hosted project of the PyTorch Foundation, under the Linux Foundation. The same year the team completed a major re-architecture known as the V1 engine, which simplified the scheduler, made features such as prefix caching and chunked prefill standard, and reduced overhead between steps.

## How it works

### Prefill and decode

Generating text has two phases. In prefill, the model processes the whole prompt at once and stores keys and values for every token in the KV cache. In decode, it produces one token at a time, each step reading the entire cache. Decode is limited mainly by memory bandwidth and by how many requests can share each step.

### Paged memory

Earlier systems reserved one contiguous memory region per request, sized for the longest possible output. Most of it went unused. vLLM splits the cache into blocks of a fixed number of tokens. Each request has a block table pointing to its blocks, which can be anywhere in GPU memory. Blocks are allocated only when needed and freed immediately when a request ends. Less waste means more requests fit at once, which raises throughput.

### Continuous batching and scheduling

At each step, the scheduler decides which requests run. Finished requests leave and waiting ones join without waiting for a whole batch to end. Chunked prefill splits long prompts into pieces so they do not stall ongoing decodes. If memory runs short, requests can be paused and resumed.

### Prefix caching

When many requests share a prefix, such as a long system prompt or document, vLLM can find the existing blocks by hashing their contents and reuse them, skipping repeated prefill work.

### Scaling out

For models too large for one GPU, tensor parallelism splits each layer across GPUs, and pipeline parallelism splits layers across stages or nodes. Mixture-of-experts models can spread experts across devices. GPUs communicate through libraries such as [NCCL](/tools/nccl).

### Using it

```python
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
params = SamplingParams(temperature=0.7, max_tokens=128)
outputs = llm.generate(["Explain paged memory in one sentence."], params)
print(outputs[0].outputs[0].text)
```

For an API server, the equivalent is `vllm serve` with the model name, which listens for OpenAI-style requests.

## Key concepts and terms

| Term | Meaning |
|---|---|
| KV cache | Stored attention keys and values for tokens already processed |
| PagedAttention | Attention computed over KV cache blocks located through a block table |
| Block | A fixed-size chunk of KV cache holding a set number of tokens |
| Continuous batching | Changing the batch membership at every generation step |
| Prefill | Processing the prompt and filling the KV cache |
| Decode | Generating output tokens one at a time |
| Chunked prefill | Splitting long prompts so they interleave with decodes |
| Prefix caching | Reusing KV blocks for shared prompt beginnings |
| Tensor parallelism | Splitting each layer's computation across GPUs |
| Time to first token | Delay before the first output token appears |
| Tokens per second | Generation throughput, per request or in total |

## A worked example

This scenario is illustrative. A legal technology firm summarises contracts with an open 8-billion-parameter model.

1. They start `vllm serve` on one 80 GB GPU with the model in bfloat16. The weights take about 16 GB, leaving most of the memory for KV cache.
2. Their existing code uses an OpenAI client. They change only the base URL and model name.
3. Every request begins with the same 1,500-token instruction block. With prefix caching on, that block is computed once and reused, cutting time to first token for typical requests from about 600 to about 250 milliseconds.
4. A load test with 64 concurrent users shows total output of several thousand tokens per second, while median time per output token stays within their target. Continuous batching keeps GPU utilisation high even though summaries vary from 100 to 900 tokens.
5. When they later move to a 70-billion-parameter model, they run it with tensor parallelism across four GPUs, with no change to client code.

## Strengths and limitations

**Strengths:**

- High throughput and efficient memory use from PagedAttention and continuous batching.
- OpenAI-compatible server makes migration from hosted APIs easy.
- Broad support for model architectures, quantisation and hardware.
- Rich features: prefix caching, speculative decoding, structured outputs, multi-LoRA.
- Open source under the Apache 2.0 licence, with neutral governance and a large community.

**Limitations:**

- Fast-moving project; configuration options and defaults change between releases.
- Tuning memory, batch sizes and parallelism still needs care for each model and GPU.
- Very large multi-node deployments need additional orchestration around it.
- It is an engine, not a full platform: authentication, routing and monitoring come from elsewhere.

## How it compares with alternatives

| Engine | Strength | Notes |
|---|---|---|
| vLLM | Broad, high-throughput general serving | Large community, wide hardware support |
| [SGLang](/tools/sglang) | Prefix reuse with RadixAttention, structured generation | Strong for agent and multi-call workloads |
| [TensorRT](/tools/tensorrt) with TensorRT-LLM | Peak performance on NVIDIA GPUs | More NVIDIA-specific build steps |
| [llama.cpp](/tools/llama-cpp) | Local and CPU inference with GGUF models | Laptops and edge, smaller scale |
| [Triton](/tools/triton) | Multi-model serving platform | Can host vLLM as a backend |

## Practical guidance and best practices

1. Start with the default settings and the OpenAI-compatible server, then tune with measurements.
2. Size GPU memory for KV cache, not only weights; long contexts and many users need much more cache.
3. Turn on prefix caching when requests share long system prompts or documents.
4. Set a maximum model length that fits your use case, which frees memory for more requests.
5. Use quantised weights such as FP8 or AWQ when memory is tight, and check output quality.
6. Measure time to first token and time per output token separately.
7. Pin the vLLM version in production and read release notes before upgrading.
8. Place a gateway in front for authentication, rate limits and routing.

## Common mistakes

- Setting the maximum context far longer than needed, which wastes cache memory.
- Benchmarking with one request at a time and missing the benefits of batching.
- Using tensor parallelism across GPUs without fast interconnects, which can slow things down.
- Mixing chat templates, so prompts are formatted differently from training.
- Upgrading without testing, then discovering changed defaults.
- Exposing the server directly to the internet without authentication.

## Frequently asked questions

### Is vLLM free?

Yes. vLLM is open source under the Apache 2.0 licence. You pay only for the hardware it runs on.

### What is PagedAttention in simple terms?

It is a way of storing each request's attention memory in small blocks, like pages in an operating system, instead of one large reserved region. Less memory is wasted, so more requests can run together.

### Can vLLM replace a hosted LLM API?

For open models, yes. Its OpenAI-compatible server lets existing client code point at it. You become responsible for hardware, scaling, security and model updates.

### Does vLLM only run on NVIDIA GPUs?

No. It supports NVIDIA and AMD GPUs and, through plug-ins, other accelerators such as Google TPUs. Feature support can vary by hardware.

### What is the difference between vLLM and SGLang?

Both are high-performance open-source LLM engines with similar core ideas. SGLang emphasises automatic prefix reuse and a programming interface for multi-step generation. Many teams benchmark both on their own workload.

## Glossary

- Inference engine: software that runs a trained model to generate outputs.
- Block table: the map from a request's tokens to KV cache blocks.
- Preemption: pausing a request to free memory for others.
- Speculative decoding: proposing several tokens cheaply and verifying them with the main model.
- LoRA adapter: small fine-tuned weights applied on top of a base model.
- Quantisation: storing weights in fewer bits to save memory.
- Chat template: the format that turns messages into the prompt a model expects.

## Summary

vLLM is an open-source LLM inference engine that achieves high throughput by managing KV cache memory in pages with PagedAttention and by batching continuously at every step. Created at Berkeley in 2023, now hosted by the PyTorch Foundation and rebuilt around its V1 engine, it offers an OpenAI-compatible server, broad model and hardware support, prefix caching and parallelism for large models. It is a strong default for self-hosting open models, provided teams size memory for the cache, measure latency carefully and add the platform pieces around it.