BentoML

Overview

BentoML is an open-source Python framework for turning trained models into production services. A developer writes a small Python class that loads a model and exposes one or more API methods. BentoML then handles the serving details: an HTTP server, request validation, adaptive batching, concurrency, packaging into a deployable unit, and building a container image. The same service can run on a laptop, in Docker, on Kubernetes, or on BentoCloud, the company's managed platform.

An everyday picture: a bento box holds a full meal in one tidy, portable container, with each part in its own compartment. BentoML packs the model, the code around it, the Python dependencies and the configuration into one unit, called a Bento, that can be carried anywhere and opened the same way each time.

BentoML is general-purpose. It serves classic models such as gradient-boosted trees, deep learning models for vision and speech, and large language models through engines such as vLLM. Its main appeal is that it keeps everything in Python while still giving production features.

What it is

Configuration such as resources, workers, timeouts and traffic limits is set in the service decorator, so the code and its runtime needs stay together.

Where it came from

BentoML was started by Chaoyu Yang, who had worked at Databricks, and was released as open source in 2019. The company behind it was founded in 2018. Early versions focused on packaging models into reproducible API servers for data science teams.

BentoML 1.0, released in July 2022, was a redesign that introduced the Bento format, a model store and a "runner" abstraction for running models in separate processes. In 2023 the team released OpenLLM for serving open language models and introduced BentoCloud. BentoML 1.2, released in February 2024, simplified the programming model: a service became a single Python class with @bentoml.service, and API methods used @bentoml.api. This is the style current documentation uses.

In February 2026 Modular announced it had acquired BentoML. The announcement stated that BentoML remains open source under the Apache 2.0 licence.

How it works

Defining a service

A service is a Python class. Its constructor loads the model once per worker. Its API methods define inputs and outputs using Python type hints, which BentoML uses to validate requests and generate an OpenAPI description.

import bentoml
from typing import List
from transformers import pipeline

@bentoml.service
class Summarization:
    def __init__(self) -> None:
        self.pipeline = pipeline("summarization")

    @bentoml.api(batchable=True)
    def summarize(self, texts: List[str]) -> List[str]:
        return [r["summary_text"] for r in self.pipeline(texts)]

bentoml serve starts the service locally with an HTTP server and a web UI for trying requests.

Adaptive batching

With batchable=True, the method receives a list of inputs. BentoML collects requests that arrive close together and passes them in as one batch, then splits the results back to each caller. Two settings matter most: max_batch_size caps how many items go into one batch, and max_latency_ms sets the longest a batch may take end to end. If requests cannot be served within that limit, the service rejects them rather than letting latency grow without bound.

Composing services

A larger system can be built from several services. One service declares another with bentoml.depends() and calls its methods as if they were local. Each service can run with different resources, so a CPU preprocessing step and a GPU model can scale independently.

Packaging and deployment

bentoml build creates a Bento with the code, model references, Python packages and system packages listed in the configuration. bentoml containerize turns it into a container image for any platform. bentoml deploy sends it to BentoCloud, which adds autoscaling, scale to zero and monitoring.

Serving LLMs

For language models, a BentoML service typically wraps an inference engine such as vLLM inside a service and exposes an OpenAI-compatible endpoint. BentoML handles packaging and deployment, while the engine handles token generation and the KV cache.

Key concepts and terms

A worked example

This scenario is illustrative. A support platform wants to classify incoming tickets by topic and urgency.

1. The team fine-tunes a small text classifier and saves it to the BentoML model store with the tag ticket-clf:3.
2. They write two services. A CPU service cleans and truncates text. A GPU service runs the classifier with batchable=True, max_batch_size=64 and max_latency_ms=200.
3. Load testing at 300 requests per second shows that batching raises GPU utilisation from about 20 per cent to about 70 per cent, and median latency stays under 80 milliseconds.
4. They run bentoml build and bentoml containerize, push the image to their registry, and deploy it to their Kubernetes cluster with a standard deployment and horizontal autoscaler.
5. When a new model version arrives, they build a new Bento, deploy it beside the old one, shift 10 per cent of traffic for a day, then complete the rollout.

Strengths and limitations

Strengths:

• Pure Python development, close to how data scientists already work.
• Built-in adaptive batching and concurrency control.
• Reproducible packaging and one-command container builds.
• Works with almost any framework, and wraps LLM engines for generative models.
• Can run anywhere, with an optional managed platform.

Limitations:

• Python overhead can matter for extremely latency-sensitive workloads.
• Large-scale traffic management on Kubernetes needs extra tooling or BentoCloud.
• Major API changes between versions, especially around 1.0 and 1.2, mean older tutorials can mislead.
• For LLMs, most performance comes from the wrapped engine, not BentoML itself.

How it compares with alternatives

Practical guidance and best practices

1. Load models in the service constructor, never inside the request method.
2. Use type hints for every API input and output, so validation and documentation are automatic.
3. Enable batching only for methods whose model benefits from it, and tune max_batch_size against memory.
4. Set max_latency_ms from your real service-level objective.
5. Split CPU and GPU work into separate services so each scales on its own.
6. Pin package versions in the Bento configuration for reproducible builds.
7. Load test with realistic traffic before production, watching latency percentiles, not only averages.
8. Follow the current documentation, since the API changed significantly in version 1.2.

Common mistakes

• Loading the model on every request, which makes latency enormous.
• Turning on batching for a model that processes one item at a time anyway.
• Setting a very high batch size that runs the GPU out of memory under load.
• Copying pre-1.2 code with runners into a current project.
• Forgetting system libraries in the build configuration, so the container fails at start.
• Measuring only average latency and missing slow tail requests.

Frequently asked questions

Is BentoML free?

Yes. The BentoML framework is open source under the Apache 2.0 licence. BentoCloud, the managed deployment platform, is a paid service.

Do I need BentoCloud to deploy BentoML services?

No. A Bento can be built into a standard container image and run on any container platform, including Kubernetes and cloud container services.

Can BentoML serve large language models?

Yes. The usual pattern is to run an engine such as vLLM inside a BentoML service and expose an OpenAI-compatible API. BentoML adds packaging, composition and deployment around the engine.

What is adaptive batching?

It is a feature that combines requests arriving at about the same time into one batch for the model, then returns each caller its own result. It raises hardware utilisation while respecting a latency limit you set.

How is BentoML different from MLflow?

MLflow focuses on tracking experiments and managing model versions. BentoML focuses on serving: turning a model into a fast, packaged API. Many teams use both.

Glossary

• Endpoint: a URL path that accepts requests for one API method.
• OpenAPI: a standard description of an HTTP API, generated from type hints.
• OCI image: the standard container image format used by Docker and Kubernetes.
• Replica: one running copy of a service.
• Throughput: the number of requests served per second.
• Tail latency: the response time of the slowest requests, such as the 99th percentile.
• Containerise: package an application and its environment into a container image.

Summary

BentoML lets developers write a model service as a plain Python class and then gives it production features: validation, adaptive batching, multi-service composition, reproducible packaging and container builds. Open source since 2019, simplified in version 1.2, and part of Modular since 2026 while staying under the Apache 2.0 licence, it suits teams who want portable model APIs without leaving Python. For the very highest throughput or Kubernetes-wide standardisation, it is often paired with or compared against specialised engines and platforms.

Related

It belongs to MLOps in my stack, under MLOps and AI infrastructure. Others in the same group: