# Serving LLMs efficiently: quantization, batching, caching, routing and adapters

Most of the cost of a popular model is serving, not training. This page covers the techniques that cut latency and cost per token.

## Overview

Serving cost depends on three things: how many bytes the model needs per token, how well the GPU is kept busy, and how often work can be reused. Each technique below targets one of them.

## The techniques

| Technique | Saves | How | Read more |
|---|---|---|---|
| Quantization | Memory and bandwidth | Store weights in 8 or 4 bits | [GPTQ](/tools/gptq), [AWQ](/tools/awq), [GGUF](/tools/gguf) |
| Continuous batching | GPU idle time | Add and remove requests token by token | [vLLM](/tools/vllm), [SGLang](/tools/sglang) |
| KV and prefix caching | Repeated computation | Reuse attention state for shared prompts | [KV cache](/tools/kv-cache), [prompt caching](/tools/prompt-caching) |
| Speculative decoding | Latency | A small model drafts, the large one verifies | [Speculative decoding](/tools/speculative-decoding) |
| Model routing | Cost per request | Send easy requests to cheap models | [LLM routing](/tools/llm-routing) |
| Adapters | Number of deployed models | Many LoRA adapters on one base model | [LoRA](/tools/lora) |
| Smaller models | Everything | Distil or choose a small model | [Small language models](/tools/small-language-models), [distillation](/tools/knowledge-distillation) |

## Quantization in practice

Eight-bit weights are nearly lossless for most models; four-bit methods such as AWQ and GPTQ keep quality close with careful calibration. The [quantization simulation](/ai-simulations/quantization) shows the error each format introduces. For laptops and edge devices, [llama.cpp](/tools/llama-cpp) runs GGUF models on CPUs and consumer GPUs.

## Memory is the main lever

Every concurrent user needs KV-cache memory, so throughput is limited by how tightly the cache is packed. Paged attention, prefix caching and, since September 2026 in vLLM, tiered offloading of the cache to CPU memory and disk let one GPU serve more users and longer contexts. The article [the LLM inference serving stack](/articles/llm-inference-serving-stack-cache-routing) explains cache-aware routing across replicas.

## Measuring

Track time to first token, tokens per second per user, throughput per GPU and cost per thousand requests. Set [cost and latency budgets](/tools/cost-and-latency-budgets) per feature and test changes against them.

## Frequently asked questions

### vLLM or SGLang?

Both are high-throughput open servers with continuous batching and prefix caching; SGLang is strong for structured and multi-call programs, vLLM for breadth of model support.

### Does quantization hurt quality?

At 8 bits rarely; at 4 bits slightly, depending on the method and model. Always evaluate on your own tasks.

## Summary

Shrink the bytes with quantization, keep the GPU busy with continuous batching, reuse work with caching, and send each request to the cheapest model that can answer it.

## Back to the topic

This page is part of **[Generative AI and LLMs](/topics/generative-ai-llms)**, which maps every concept, tool, simulation and article on the subject.
