# Ray Serve

Ray Serve is a scalable serving library built on Ray for deploying models and multi-model pipelines in Python, with autoscaling and LLM support.

## Overview

Ray Serve is a model serving library built on Ray, an open-source framework for distributed Python. It lets developers turn Python classes and functions into scalable web services, called deployments, and connect several of them into one application. Ray Serve handles replicas, load balancing, autoscaling, batching and fault tolerance across a cluster of machines, while the developer keeps writing ordinary Python.

An everyday picture: a busy café with several stations. One barista takes orders, two make coffee, one makes sandwiches. When a queue forms at the coffee station, the manager moves a third person there. Customers see one counter, but behind it work is split, routed and rebalanced. Ray Serve is that manager for model services. Each station is a deployment with its own staff count, and the system shifts capacity to where demand is.

Ray Serve is especially strong when an application uses several models or mixes model calls with business logic, such as retrieval, ranking and generation in one request. It also has a dedicated layer for serving large language models.

## What it is

| Part | Purpose |
|---|---|
| Deployment | A Python class or function marked with `@serve.deployment`, run as a pool of replicas |
| Replica | One running copy of a deployment, implemented as a Ray actor |
| Application | A group of deployments started together with `serve.run` |
| Ingress deployment | The deployment that receives external HTTP or gRPC traffic, optionally using FastAPI |
| DeploymentHandle | A Python object that lets one deployment call another |
| Controller | The process that manages deployments and recovers from failures |
| Proxy | The HTTP and gRPC entry point that routes requests to replicas |
| Autoscaler | Adjusts replica counts based on ongoing requests |
| Ray Serve LLM | An add-on for serving language models with OpenAI-compatible APIs |

Because it runs on Ray, Serve can use any resources Ray manages: CPUs, GPUs, fractional GPUs and custom resources, across many nodes.

## Where it came from

Ray was created at the University of California, Berkeley, in the RISELab, by researchers including Robert Nishihara and Philipp Moritz working with Ion Stoica. It was described in a paper at the OSDI conference in 2018. The team later founded Anyscale to develop Ray and offer a managed platform.

Ray Serve grew out of work in 2019 and shipped as a supported library with Ray 1.0 in September 2020. Ray 2.0 in 2022 brought a more stable composition API, and later releases added support for running several applications on one cluster and model multiplexing. In 2025 the project introduced Ray Serve LLM for language model serving. In October 2025 Ray joined the PyTorch Foundation, part of the Linux Foundation, giving it neutral governance alongside [PyTorch](/tools/pytorch) and [vLLM](/tools/vllm).

## How it works

### Writing a deployment

A deployment is a class with a decorator. Its constructor loads models; its `__call__` method handles HTTP requests.

```python
from ray import serve

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Classifier:
    def __init__(self):
        self.model = lambda x: len(x) % 2  # load a real model here

    async def __call__(self, request) -> dict:
        data = await request.json()
        return {"label": self.model(data["text"])}

app = Classifier.bind()
serve.run(app)
```

`bind()` builds the application graph without starting it. `serve.run()` starts it on the Ray cluster.

### Composition

Deployments can receive handles to other deployments when they are bound. Calling a handle returns a response object that can be awaited, so one request can fan out to several models in parallel and combine their results. Each deployment scales independently. A cheap text cleaner might run on four CPU replicas while an expensive model runs on two GPU replicas.

### Autoscaling

Autoscaling is configured per deployment with settings such as `min_replicas`, `max_replicas` and `target_ongoing_requests`, the average number of in-flight requests each replica should handle. When load rises above the target, Serve adds replicas; when it falls, it removes them. If Ray's cluster autoscaler is enabled, new replicas can trigger new machines.

### Batching and multiplexing

The `@serve.batch` decorator groups individual calls into batches for models that benefit from vectorised execution. Model multiplexing lets many small models, such as one per customer, share a pool of replicas and be loaded on demand.

### Serving language models

Ray Serve LLM wraps engines such as vLLM in Serve deployments and exposes OpenAI-compatible endpoints. It supports multiple LoRA adapters on one base model, prefix-aware routing to reuse the [KV cache](/tools/kv-cache), separate scaling of prefill and decode phases, and models spread over several nodes.

### Production operation

In production, applications are usually described in a YAML configuration and deployed with the `serve deploy` command or on Kubernetes through KubeRay, an operator for running Ray clusters. Configuration changes can be applied in place, and the controller restarts failed replicas automatically.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Ray | A framework for distributed Python using tasks and actors |
| Actor | A stateful Ray worker process |
| Deployment | A scalable unit of serving code with its own replicas |
| Replica | One running instance of a deployment |
| Bind | Build an application graph from deployments and arguments |
| DeploymentHandle | A handle used to call another deployment |
| Ingress | The deployment that receives external requests |
| `target_ongoing_requests` | Desired average in-flight requests per replica |
| `@serve.batch` | Decorator that groups requests into batches |
| Multiplexing | Serving many models from a shared replica pool |
| KubeRay | The operator that runs Ray clusters on Kubernetes |

## A worked example

This scenario is illustrative. An online marketplace builds a search service that ranks listings and writes a short explanation.

1. The ingress deployment, written with FastAPI, receives a query and calls three deployments: an embedding model, a ranking model and a small language model.
2. The embedding model runs on two GPU replicas with `@serve.batch`, grouping up to 32 queries. The ranker runs on six CPU replicas. The language model runs through Ray Serve LLM on one GPU per replica.
3. Autoscaling is set to `min_replicas=1`, `max_replicas=8` and `target_ongoing_requests=4` for the language model. At the evening peak it scales to seven replicas; overnight it returns to one.
4. Embedding and ranking run in parallel through handles, cutting median latency from 410 to 260 milliseconds compared with calling them one after another.
5. A new ranking model is rolled out by updating the YAML configuration. Serve starts new replicas, shifts traffic and stops the old ones without downtime.

## Strengths and limitations

**Strengths:**

- Pure Python, with natural composition of many models and business logic.
- Independent scaling and resource settings for each deployment.
- Uses Ray's cluster, so the same infrastructure can run training, data processing and serving.
- Strong features for LLM serving, including multi-LoRA and prefix-aware routing.
- Open source under the Apache 2.0 licence, with neutral governance.

**Limitations:**

- Running a Ray cluster is another system to learn and operate.
- Python overhead can limit the very lowest latencies.
- Debugging distributed actors can be harder than debugging one web server.
- For a single simple model, it may be more machinery than needed.

## How it compares with alternatives

| Tool | Approach | Best fit |
|---|---|---|
| Ray Serve | Python deployments on a Ray cluster | Multi-model pipelines and Python-heavy logic |
| [BentoML](/tools/bentoml) | Python services packaged as containers | Portable single or few-model APIs |
| [KServe](/tools/kserve) | Kubernetes custom resources | Platform-wide standard serving |
| [Triton](/tools/triton) | High-performance inference server | Maximum throughput on optimised models |
| [vLLM](/tools/vllm) | Dedicated LLM engine | Single LLM endpoints; often used inside Ray Serve |

## Practical guidance and best practices

1. Split work into deployments by resource type, such as CPU pre-processing and GPU inference.
2. Load models in constructors, not request handlers.
3. Set `max_ongoing_requests` to protect replicas from overload.
4. Tune autoscaling targets with load tests rather than guesses.
5. Use `@serve.batch` only where the model benefits from batching.
6. Describe production applications in YAML and keep it in version control.
7. Run on Kubernetes through KubeRay if your organisation already uses Kubernetes.
8. Export Ray and Serve metrics to your monitoring stack and watch queue lengths.

## Common mistakes

- Putting every model in one deployment, so nothing can scale independently.
- Calling deployments one after another when they could run in parallel.
- Forgetting GPU options in `ray_actor_options`, so models run on CPU.
- Setting `min_replicas` to zero for large models with slow start-up.
- Ignoring head node resilience, which matters for long-running clusters.
- Treating local tests on one machine as proof of cluster behaviour.

## Frequently asked questions

### Do I need a Ray cluster to use Ray Serve?

Ray Serve runs on Ray, but a cluster can be a single machine for development. In production it usually runs on several machines, often on Kubernetes through KubeRay or on a managed Ray service.

### Can Ray Serve use FastAPI?

Yes. An ingress deployment can wrap a FastAPI app, so routes, validation and documentation work as usual while Serve handles scaling.

### How does Ray Serve serve LLMs?

Ray Serve LLM runs engines such as vLLM inside deployments and adds OpenAI-compatible APIs, autoscaling, multi-LoRA serving and routing that reuses cached prompt prefixes.

### Is Ray Serve only for Anyscale customers?

No. Ray and Ray Serve are open source and can run anywhere. Anyscale offers a managed platform, but it is optional.

### When should I choose Ray Serve over KServe?

Choose Ray Serve when an application combines several models and Python logic in one request flow. Choose KServe when the priority is a uniform, Kubernetes-native way to deploy many independent models. Some teams use both.

## Glossary

- Head node: the Ray node that runs cluster management processes.
- Worker node: a Ray node that runs tasks and actors.
- Fractional GPU: a share of one GPU assigned to a replica.
- Back-pressure: rejecting or queueing requests when replicas are full.
- LoRA adapter: a small set of fine-tuned weights applied to a base model.
- Prefix-aware routing: sending requests with shared prompt beginnings to the same replica.
- Cluster autoscaler: the Ray component that adds or removes machines.

## Summary

Ray Serve turns Python classes into scalable deployments that can be composed into multi-model applications, with per-deployment resources, autoscaling, batching and fault tolerance on a Ray cluster. Born from Berkeley's RISELab, shipped with Ray 1.0 in 2020 and part of the PyTorch Foundation since 2025, it now includes a dedicated layer for serving language models. It is a strong fit for teams whose serving logic is more than one model behind one endpoint, provided they are ready to run Ray.