# KServe

KServe is an open-source Kubernetes platform for serving ML and generative AI models with autoscaling, canary rollouts and standard inference protocols.

## Overview

KServe is an open-source platform for serving machine learning and generative AI models on [Kubernetes](/tools/kubernetes). Instead of writing a web server, a container definition, an autoscaler and a load balancer for each model, a team writes one short resource called an `InferenceService` that says where the model is stored and what format it uses. KServe then creates the serving containers, exposes an endpoint, scales the replicas with traffic and manages rollouts of new versions.

An everyday picture: think of a restaurant franchise. A new branch does not design its own kitchen, staffing plan and ordering system. It fills in a standard form, and head office supplies a proven kitchen layout, trained staff and a till that works like every other branch. KServe is that franchise system for models. Each model fills in a standard form, and the platform supplies a consistent, production-ready setup.

KServe is popular with platform teams who want every model in an organisation, from a scikit-learn classifier to a large language model, to be deployed, observed and governed in the same way.

## What it is

| Part | Purpose |
|---|---|
| InferenceService | The main custom resource describing a model deployment |
| Predictor | The component that loads the model and returns predictions |
| Transformer | Optional pre- and post-processing around the predictor |
| Explainer | Optional component returning explanations for predictions |
| Serving runtimes | Reusable templates for model servers, such as Triton, vLLM or the scikit-learn server |
| Storage initialiser | Downloads model files from object storage or other sources at start-up |
| Inference protocols | Standard request formats, including the Open Inference Protocol and OpenAI-compatible APIs for generative models |
| ModelMesh | A mode for packing many small models densely onto shared servers |
| LLMInferenceService | A resource for large language model serving with features such as prefill and decode separation |

KServe can run in two broad deployment styles. In serverless mode it builds on Knative, which provides request-based autoscaling and scale to zero. In standard or raw deployment mode it uses ordinary Kubernetes deployments and autoscalers, which is simpler to operate and common for GPU-heavy generative workloads.

## Where it came from

KServe began in 2019 as KFServing, part of the [Kubeflow](/tools/kubeflow) project. It was a collaboration among engineers from Google, IBM, Bloomberg, NVIDIA and Seldon, who each needed a standard way to serve models on Kubernetes.

In September 2021 the project was renamed KServe and became independent of Kubeflow, while remaining closely integrated with it. In early 2022 it was donated to the LF AI and Data Foundation. Over the following years it added support for generative AI, including a vLLM-based backend, OpenAI-compatible endpoints and integration with distributed LLM serving work. In 2025 KServe was accepted by the Cloud Native Computing Foundation as an incubating project.

## How it works

### Declaring a model

A minimal `InferenceService` names the model format and its storage location.

```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
```

When this is applied, the KServe controller matches the format to a serving runtime, creates pods, runs the storage initialiser to fetch the model, and exposes an HTTP endpoint.

### Serving runtimes

Runtimes separate "what model" from "which server". A cluster administrator defines runtimes for formats such as scikit-learn, XGBoost, ONNX, TensorFlow, PyTorch and Hugging Face models, often backed by servers such as [Triton](/tools/triton) or [vLLM](/tools/vllm). Model owners then only declare the format.

### Autoscaling

In serverless mode, Knative scales replicas on concurrent requests and can scale to zero when idle, waking the service on the next request. In standard mode, the Kubernetes horizontal pod autoscaler or an event-driven autoscaler scales on metrics such as CPU, GPU or queue length. Large models usually keep at least one replica warm, because cold starts can take minutes.

### Rollouts

When a new model version is applied, KServe can send a set share of traffic to it, using the `canaryTrafficPercent` field, while the rest goes to the previous revision. If metrics look good, the share is raised to 100; if not, it is rolled back.

### Pre-processing, post-processing and pipelines

A transformer runs before and after the predictor, for example to turn raw text into features or to format outputs. Several models can be chained into an inference graph for multi-step flows.

### Generative AI

For language models, KServe exposes OpenAI-compatible endpoints, loads models from Hugging Face or storage, and works with features such as [KV cache](/tools/kv-cache) reuse and multi-node serving for models too large for one machine.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Custom resource definition | A Kubernetes extension that adds a new object type |
| InferenceService | The KServe object describing one model deployment |
| Predictor | The part that holds the model and computes outputs |
| Transformer | Optional processing before and after prediction |
| ServingRuntime | A template describing a model server and the formats it supports |
| Storage URI | The address of model files, such as an object storage path |
| Knative | A Kubernetes add-on that provides request-driven autoscaling |
| Scale to zero | Removing all replicas when there is no traffic |
| Canary rollout | Sending a small share of traffic to a new version first |
| Open Inference Protocol | A standard REST and gRPC request format shared by several servers |
| ModelMesh | A high-density mode for serving many models on shared pods |

## A worked example

This scenario is illustrative. An insurance company runs 40 models, from fraud scores to document classifiers, and wants one serving approach.

1. The platform team installs KServe on its Kubernetes cluster in standard deployment mode, with serving runtimes for XGBoost, ONNX and Hugging Face models.
2. The fraud team stores its model in object storage and applies a 12-line InferenceService. Within minutes it has an endpoint at a predictable internal address.
3. Autoscaling keeps two replicas at night and grows to eight during the morning peak of about 1,200 requests per second.
4. A retrained version is deployed with `canaryTrafficPercent: 10`. Dashboards show the new version's error rate and latency next to the old one for 48 hours. The team then moves to 100 per cent.
5. A document team deploys a 7-billion-parameter language model with the vLLM runtime on one GPU node, using the OpenAI-compatible endpoint so existing client code works unchanged.
6. All 40 services report the same metrics into [Prometheus](/tools/prometheus) and [Grafana](/tools/grafana), so on-call engineers use one set of dashboards.

## Strengths and limitations

**Strengths:**

- One declarative, Kubernetes-native way to serve many kinds of models.
- Pluggable runtimes, including high-performance servers.
- Autoscaling, scale to zero and canary rollouts built in.
- Standard inference protocols reduce client lock-in.
- Open governance under the CNCF, with broad industry backing.

**Limitations:**

- Requires a working Kubernetes platform and the skills to run it.
- Serverless mode adds Knative and networking components to operate.
- Cold starts for large models make scale to zero impractical for many LLM uses.
- Debugging spans several layers: controller, runtime, networking and storage.
- Small teams with a few models may find it heavier than a simple container deployment.

## How it compares with alternatives

| Tool | Approach | Best fit |
|---|---|---|
| KServe | Kubernetes resources for model serving | Platform teams standardising many models |
| [BentoML](/tools/bentoml) | Python services packaged as containers | Developer-friendly APIs, any platform |
| [Ray Serve](/tools/ray-serve) | Python deployments on Ray | Composed multi-model Python applications |
| [Triton](/tools/triton) | High-performance inference server | Raw throughput; often used inside KServe |
| [SageMaker](/tools/sagemaker) endpoints | Managed cloud hosting | Teams preferring managed AWS services |

## Practical guidance and best practices

1. Choose standard deployment mode for large GPU models unless you truly need scale to zero.
2. Define a small set of approved serving runtimes and let teams declare only the model format.
3. Set resource requests and limits for every predictor, including GPU counts.
4. Keep a minimum replica count for latency-sensitive services to avoid cold starts.
5. Use canary rollouts with clear success metrics and a time limit.
6. Cache or pre-load large model files near the cluster to cut start-up time.
7. Export metrics and traces from every service to one monitoring stack.
8. Version InferenceService definitions in git and apply them through a GitOps tool such as [Argo CD](/tools/argocd).

## Common mistakes

- Enabling scale to zero for a large language model and then wondering why first requests time out.
- Forgetting storage credentials, so the storage initialiser fails and pods never become ready.
- Skipping readiness checks, so traffic reaches pods before the model is loaded.
- Running canaries without comparing metrics, which makes them pointless.
- Letting each team write its own custom runtime instead of reusing shared ones.
- Under-sizing memory for model downloads and unpacking.

## Frequently asked questions

### Is KServe the same as KFServing?

Yes. KFServing was the original name while it was part of Kubeflow. It was renamed KServe in 2021 when it became an independent project.

### Do I need Kubeflow to use KServe?

No. KServe installs on any Kubernetes cluster by itself. It integrates with Kubeflow but does not depend on it.

### Can KServe serve large language models?

Yes. It has runtimes for engines such as vLLM, OpenAI-compatible endpoints and features for large, multi-GPU deployments. It handles the Kubernetes side, while the engine handles generation.

### What is the difference between serverless and standard mode?

Serverless mode uses Knative for request-driven autoscaling and scale to zero. Standard mode uses plain Kubernetes deployments and autoscalers, which is simpler and common for always-on GPU services.

### Does KServe lock me into one model server?

No. Serving runtimes are pluggable, and the Open Inference Protocol is shared with other servers, so clients and models can move between runtimes.

## Glossary

- Controller: the KServe program that turns InferenceService objects into running pods.
- Revision: one version of a deployed service, used for traffic splitting.
- Cold start: the delay while a new replica starts and loads its model.
- Readiness probe: a check that tells Kubernetes when a pod can take traffic.
- Inference graph: a chain of models and steps served as one flow.
- gRPC: a fast binary protocol supported alongside REST.
- LF AI and Data: a Linux Foundation umbrella that hosted KServe before the CNCF.

## Summary

KServe gives Kubernetes a standard, declarative way to serve models: an InferenceService names the model and format, and the platform handles servers, scaling, rollouts and endpoints. It began as KFServing in Kubeflow in 2019, became KServe in 2021 and joined the CNCF as an incubating project in 2025, gaining strong generative AI support along the way. It is a good choice for organisations that run Kubernetes and want every model served the same way, provided they have the platform skills to operate it.