Serving LLMs efficiently: quantization, batching, caching, routing and adapters
Most of the cost of a popular model is serving, not training. This page covers the techniques that cut latency and cost per token.
Overview
Serving cost depends on three things: how many bytes the model needs per token, how well the GPU is kept busy, and how often work can be reused. Each technique below targets one of them.
The techniques
| Technique | Saves | How | Read more |
|---|---|---|---|
| Quantization | Memory and bandwidth | Store weights in 8 or 4 bits | GPTQ, AWQ, GGUF |
| Continuous batching | GPU idle time | Add and remove requests token by token | vLLM, SGLang |
| KV and prefix caching | Repeated computation | Reuse attention state for shared prompts | KV cache, prompt caching |
| Speculative decoding | Latency | A small model drafts, the large one verifies | Speculative decoding |
| Model routing | Cost per request | Send easy requests to cheap models | LLM routing |
| Adapters | Number of deployed models | Many LoRA adapters on one base model | LoRA |
| Smaller models | Everything | Distil or choose a small model | Small language models, distillation |
Quantization in practice
Eight-bit weights are nearly lossless for most models; four-bit methods such as AWQ and GPTQ keep quality close with careful calibration. The quantization simulation shows the error each format introduces. For laptops and edge devices, llama.cpp runs GGUF models on CPUs and consumer GPUs.
Memory is the main lever
Every concurrent user needs KV-cache memory, so throughput is limited by how tightly the cache is packed. Paged attention, prefix caching and, since September 2026 in vLLM, tiered offloading of the cache to CPU memory and disk let one GPU serve more users and longer contexts. The article the LLM inference serving stack explains cache-aware routing across replicas.
Measuring
Track time to first token, tokens per second per user, throughput per GPU and cost per thousand requests. Set cost and latency budgets per feature and test changes against them.
Frequently asked questions
vLLM or SGLang?
Both are high-throughput open servers with continuous batching and prefix caching; SGLang is strong for structured and multi-call programs, vLLM for breadth of model support.
Does quantization hurt quality?
At 8 bits rarely; at 4 bits slightly, depending on the method and model. Always evaluate on your own tasks.
Summary
Shrink the bytes with quantization, keep the GPU busy with continuous batching, reuse work with caching, and send each request to the cheapest model that can answer it.
Back to the topic
This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.