Özer Cem KELAHMETKELAHMET
Topics · Generative AI and LLMs

Serving LLMs efficiently: quantization, batching, caching, routing and adapters

Most of the cost of a popular model is serving, not training. This page covers the techniques that cut latency and cost per token.

Overview

Serving cost depends on three things: how many bytes the model needs per token, how well the GPU is kept busy, and how often work can be reused. Each technique below targets one of them.

The techniques

Technique Saves How Read more
Quantization Memory and bandwidth Store weights in 8 or 4 bits GPTQ, AWQ, GGUF
Continuous batching GPU idle time Add and remove requests token by token vLLM, SGLang
KV and prefix caching Repeated computation Reuse attention state for shared prompts KV cache, prompt caching
Speculative decoding Latency A small model drafts, the large one verifies Speculative decoding
Model routing Cost per request Send easy requests to cheap models LLM routing
Adapters Number of deployed models Many LoRA adapters on one base model LoRA
Smaller models Everything Distil or choose a small model Small language models, distillation

Quantization in practice

Eight-bit weights are nearly lossless for most models; four-bit methods such as AWQ and GPTQ keep quality close with careful calibration. The quantization simulation shows the error each format introduces. For laptops and edge devices, llama.cpp runs GGUF models on CPUs and consumer GPUs.

Memory is the main lever

Every concurrent user needs KV-cache memory, so throughput is limited by how tightly the cache is packed. Paged attention, prefix caching and, since September 2026 in vLLM, tiered offloading of the cache to CPU memory and disk let one GPU serve more users and longer contexts. The article the LLM inference serving stack explains cache-aware routing across replicas.

Measuring

Track time to first token, tokens per second per user, throughput per GPU and cost per thousand requests. Set cost and latency budgets per feature and test changes against them.

Frequently asked questions

vLLM or SGLang?

Both are high-throughput open servers with continuous batching and prefix caching; SGLang is strong for structured and multi-call programs, vLLM for breadth of model support.

Does quantization hurt quality?

At 8 bits rarely; at 4 bits slightly, depending on the method and model. Always evaluate on your own tasks.

Summary

Shrink the bytes with quantization, keep the GPU busy with continuous batching, reuse work with caching, and send each request to the cheapest model that can answer it.

Back to the topic

This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.

Last updated Nov 18, 2026
Type to search.
    ↑↓ navigate↵ open