NVIDIA Triton Inference Server: multi-framework model serving

Overview

NVIDIA Triton Inference Server is open-source software for serving trained models in production. It loads models from many frameworks, including TensorRT, PyTorch, ONNX Runtime, TensorFlow, OpenVINO, Python code and tree-based models, and exposes them through standard HTTP and gRPC endpoints. Its strength is squeezing high throughput from GPUs and CPUs, using features such as dynamic batching, running several model copies at once and chaining models into pipelines.

An everyday picture: a busy airport with one control tower handling planes from many airlines. Each airline flies different aircraft, but the tower uses one set of rules to schedule runways, group arrivals and keep traffic moving. Triton is that control tower for models. Each framework is a different airline; Triton schedules their requests on shared hardware with one consistent interface.

Since 2025 NVIDIA has presented Triton as part of its Dynamo inference platform, under the name NVIDIA Dynamo-Triton. The server itself, its repository and its configuration format remain the same, and most documentation and tools still call it Triton.

What it is

Part	Purpose
Model repository	A directory or object storage location holding models, versions and configuration
Model configuration	A `config.pbtxt` file describing inputs, outputs, batching and instances
Backends	Plug-ins that execute models for each framework, such as TensorRT, PyTorch, ONNX Runtime, Python, vLLM and TensorRT-LLM
Scheduler and batcher	Queue requests and combine them into batches
Instance groups	Run several copies of a model on chosen GPUs or CPUs
Ensembles	Chain models and processing steps into one served pipeline
Business Logic Scripting	Python code that calls other models with loops and conditions
Endpoints	HTTP and gRPC APIs following the KServe inference protocol, plus metrics
Model Analyzer and perf_analyzer	Tools to measure throughput and latency and find good configurations

Triton is released under the BSD 3-clause licence and is commonly run from NVIDIA's prebuilt containers, which are updated roughly monthly.

Where it came from

NVIDIA first released the software around 2018 as the TensorRT Inference Server, focused on serving TensorRT-optimised models on its GPUs. Users wanted one server for every framework, so support for TensorFlow, PyTorch, ONNX and others was added, and in 2020 it was renamed Triton Inference Server to reflect that it was no longer tied to TensorRT.

Over the following years Triton gained ensembles, Business Logic Scripting, a Python backend, a forest inference backend for gradient-boosted trees, sequence batching for stateful models, and backends for large language model engines such as TensorRT-LLM and vLLM. In March 2025 NVIDIA announced Dynamo, a framework for distributed LLM inference, and grouped Triton with it under the Dynamo name.

How it works

The model repository

Triton reads models from a repository with a fixed layout: one folder per model, containing a config.pbtxt file and numbered version subfolders such as 1/ and 2/ that hold the model files. A version policy decides which versions are served, by default the latest. Models can be loaded at start-up or on request through a model control API.

Model configuration

The configuration names the backend, the inputs and outputs with types and shapes, the maximum batch size and scheduling options. A minimal example with dynamic batching and two instances per GPU:

backend: "onnxruntime"
max_batch_size: 64
dynamic_batching { max_queue_delay_microseconds: 2000 }
instance_group [ { count: 2, kind: KIND_GPU } ]

Dynamic batching

Individual requests arrive one at a time. The dynamic batcher waits a small, configured time to collect several, runs them as one batch, then returns each result to its caller. GPUs are much more efficient on batches, so throughput can rise several times while latency grows only by the short wait.

Concurrent execution

Instance groups run several copies of a model, on one GPU or across GPUs. While one copy waits for data transfer, another computes, which keeps hardware busy. Different models can also share the same GPU.

Pipelines

An ensemble connects models and steps as a graph, for example tokeniser, encoder and classifier, so a client makes one call. Business Logic Scripting allows more flexible flows written in Python.

Serving LLMs

For language models, Triton uses the TensorRT-LLM or vLLM backend. These handle continuous batching and KV cache management, while Triton provides the endpoints, metrics and multi-model hosting.

Observability and tuning

Triton exposes metrics in Prometheus format, including request counts, queue time and GPU use. perf_analyzer generates load to measure latency and throughput, and Model Analyzer searches batch sizes and instance counts to recommend configurations.

Key concepts and terms

Term	Meaning
Model repository	The storage location and layout Triton loads models from
`config.pbtxt`	A model's configuration file in protocol buffer text format
Backend	The component that runs models of a given framework
Dynamic batching	Server-side grouping of separate requests into batches
Queue delay	The longest time a request may wait to be batched
Instance group	The number and placement of model copies
Ensemble	A served graph of models and steps
Business Logic Scripting	Python code that orchestrates calls to other models
Sequence batching	Batching for stateful models that keep context between requests
KServe protocol	A standard inference API also used by other servers
perf_analyzer	A load generator for measuring Triton performance

A worked example

This scenario is illustrative. A payments company runs three models for each card transaction: a feature encoder, a gradient-boosted fraud model and a small neural risk model.

The team converts the neural model to ONNX and the tree model to a format for the forest inference backend, and places all three in a model repository.
An ensemble passes raw features through the encoder, sends the result to both models in parallel and returns both scores in one call.
Without batching, one T4-class GPU handles about 1,800 requests per second at a median of 4 milliseconds. With dynamic batching at a 1 millisecond queue delay and two instances of the neural model, perf_analyzer shows about 6,500 requests per second at a median of 6 milliseconds, within the 20 millisecond budget.
Triton runs as a container on Kubernetes, with autoscaling driven by queue time from the metrics endpoint.
A new fraud model version is copied into folder 2, loaded through the model control API, tested, then made the served version.

Strengths and limitations

Strengths:

One server for many frameworks and model types, including trees and LLMs.
High GPU utilisation through dynamic batching and concurrent instances.
Pipelines with ensembles and Business Logic Scripting avoid extra network hops.
Standard HTTP and gRPC APIs and built-in metrics.
Open source, with tuning tools and frequent releases.

Limitations:

Configuration files and repository layout take time to learn.
Best performance usually requires converting models to optimised formats.
Python-heavy custom logic is less natural than in Python-first servers.
Large containers and many backends add operational weight.
For pure LLM serving, a dedicated engine alone may be simpler.

How it compares with alternatives

Tool	Focus	When to choose
Triton	Multi-framework, high-throughput serving	Many model types, maximum GPU efficiency
vLLM	LLM inference engine	Mainly language models
KServe	Kubernetes serving platform	Standardising deployments; can run Triton inside
BentoML	Python services and packaging	Developer-friendly Python APIs
Ray Serve	Python deployments on Ray	Complex Python composition

Practical guidance and best practices

Start with a working model in its native backend, then optimise with TensorRT or ONNX Runtime and measure the gain.
Enable dynamic batching for stateless models and set a queue delay that fits your latency budget.
Use perf_analyzer or Model Analyzer to choose batch sizes and instance counts.
Set max_batch_size from measured memory use, not guesses.
Use ensembles for multi-step pipelines to avoid extra network calls.
Pin container versions and test upgrades, since releases are frequent.
Scrape metrics and alert on queue time as well as latency.
Keep the model repository in object storage with versioned folders.

Common mistakes

Declaring inputs and outputs with wrong shapes or types, so models fail to load.
Setting a long queue delay and then missing latency targets.
Running many instances of a large model and exhausting GPU memory.
Using the Python backend for heavy compute that should run in an optimised backend.
Benchmarking with single requests and drawing conclusions about batched production load.
Forgetting warm-up, so first requests after start are slow.

Frequently asked questions

Is Triton only for NVIDIA GPUs?

No. Triton runs models on CPUs as well, including through OpenVINO and ONNX Runtime, and is used on x86 and Arm servers. Its strongest optimisations target NVIDIA GPUs.

What is the difference between Triton and TensorRT?

TensorRT is a compiler and runtime that optimises a model for NVIDIA GPUs. Triton is a server that hosts models, including TensorRT engines, and handles requests, batching and scheduling.

Can Triton serve large language models?

Yes, through backends such as TensorRT-LLM and vLLM. Those engines handle token generation; Triton adds endpoints, metrics and multi-model hosting.

What happened to the name Triton?

In 2025 NVIDIA grouped Triton with its Dynamo inference platform and began calling it NVIDIA Dynamo-Triton. The software and configuration format continue, and the name Triton is still widely used.

Does Triton work with KServe?

Yes. Triton follows the KServe inference protocol and is available as a serving runtime in KServe, so it can be deployed through KServe's InferenceService resource.

Glossary

Inference: running a trained model to produce outputs.
Throughput: the number of requests or items processed per second.
Latency: the time to answer one request.
gRPC: a fast binary remote procedure call protocol.
Model control API: endpoints to load and unload models at run time.
Forest inference: fast execution of tree ensembles such as XGBoost on GPUs.
Warm-up: sending sample requests at start-up so first real requests are fast.

Summary

Triton Inference Server hosts models from many frameworks behind one standard API and makes hardware work hard through dynamic batching, concurrent instances and in-server pipelines. Begun around 2018 as the TensorRT Inference Server, renamed Triton in 2020 and grouped with NVIDIA Dynamo in 2025, it remains open source and widely used inside Kubernetes platforms such as KServe. It rewards teams that invest in configuration and measurement, and it fits best where many models must be served efficiently on shared GPUs.

It belongs to Inference and serving in my stack, under MLOps and AI infrastructure. Others in the same group:

vLLM SGLang TensorRT ONNX Runtime llama.cpp