# Model serving: inference servers, autoscaling and LLM serving engines

Serving turns a model into an API with latency and cost targets. This page covers inference servers for classic models and the specialised engines for LLMs.

## Overview

Classic models are small and fast; the challenge is scale and reliability. LLMs are large and generate token by token; the challenge is memory and throughput. Both need autoscaling, monitoring and clear latency budgets.

## Serving options

| Need | Options | Read more |
|---|---|---|
| Classic models as APIs | BentoML, KServe, Ray Serve, SageMaker endpoints | [BentoML](/tools/bentoml), [KServe](/tools/kserve), [Ray Serve](/tools/ray-serve) |
| Optimised GPU inference | TensorRT, Triton, ONNX Runtime | [TensorRT](/tools/tensorrt), [ONNX Runtime](/tools/onnx-runtime) |
| Open-weight LLMs at scale | vLLM, SGLang | [vLLM](/tools/vllm), [SGLang](/tools/sglang) |
| LLMs on CPU or edge | llama.cpp, ExLlamaV2 | [llama.cpp](/tools/llama-cpp), [ExLlamaV2](/tools/exllamav2) |
| Managed APIs | Cloud model APIs | [Vertex AI](/tools/vertex-ai), [SageMaker](/tools/sagemaker) |

## What makes LLM serving different

Each request holds KV-cache memory for its whole context, so the number of concurrent users is set by GPU memory. Engines use paged attention, continuous batching and prefix caching to pack requests tightly, and offload cold cache to CPU memory or disk. Quantised weights free memory for more cache. See [serving and cost](/topics/generative-ai-llms/serving-and-cost) and the [KV cache simulation](/ai-simulations/kv-cache).

## Autoscaling and routing

Scale on queue length and GPU utilisation rather than CPU; keep warm replicas because loading a large model takes minutes; route requests with shared prefixes to the same replica to reuse cache; and send easy requests to smaller models with [LLM routing](/tools/llm-routing).

## Budgets and resilience

Set targets for time to first token, tokens per second and cost per request. Add timeouts, retries with back-off and a [circuit breaker](/tools/circuit-breaker) in front of model calls, and a fallback model for outages.

## Frequently asked questions

### GPU or CPU for classic models?

CPU is usually enough for trees and small networks; GPUs pay off for deep networks, high throughput or LLMs.

### Self-host or use an API?

APIs are simplest and track the newest models; self-hosting open weights gives control over data, latency and cost at high volume.

## Summary

Pick the server by model type, plan LLM serving around KV-cache memory, autoscale on queue and GPU load, and protect every model call with budgets, timeouts and fallbacks.

## Back to the topic

This page is part of **[MLOps and AI infrastructure](/topics/mlops-ai-infrastructure)**, which maps every concept, tool, simulation and article on the subject.
