# NVIDIA TensorRT

TensorRT is NVIDIA's SDK for optimising trained models into fast inference engines with layer fusion, precision calibration and kernel tuning. Learn how it works.

## Overview

NVIDIA TensorRT is a software development kit for making trained neural networks run fast on NVIDIA GPUs. It takes a model, usually exported from [PyTorch](/tools/pytorch) or TensorFlow in the ONNX format, analyses its graph, and builds an optimised "engine" for a specific GPU. The engine fuses operations, chooses the fastest kernels, uses lower numerical precision where it is safe and plans memory carefully. At run time, a small runtime library loads the engine and executes it with very little overhead.

An everyday picture: a delivery company planning tomorrow's routes. One approach sends drivers out and lets them decide turns as they go. A better approach studies the whole map the night before, combines nearby stops, picks the fastest roads for each van type and loads parcels in the right order. TensorRT does that planning for a model. The work happens once, at build time, so each prediction afterwards follows the best route.

TensorRT has a companion library for large language models, TensorRT-LLM, which adds the features LLM serving needs, such as paged [KV cache](/tools/kv-cache) management and in-flight batching. Both are commonly served through [Triton](/tools/triton).

## What it is

| Part | Purpose |
|---|---|
| Parser | Imports a model graph, mainly from ONNX |
| Builder | Optimises the graph and produces an engine for a target GPU |
| Engine (plan) | The serialised, optimised model ready to run |
| Runtime | Loads engines and executes inference |
| Layer and tensor fusion | Combines several operations into one kernel |
| Precision modes | FP32, FP16, BF16, INT8, FP8 and, on newer GPUs, FP4 |
| Kernel auto-tuning | Tests candidate kernels and picks the fastest for each layer |
| Optimisation profiles | Define ranges of input shapes for dynamic sizes |
| Plug-ins | Custom layers for operations TensorRT does not support natively |
| trtexec | Command-line tool to build engines and benchmark them |
| TensorRT-LLM | Library and runtime specialised for large language models |
| Model Optimizer | NVIDIA tooling for quantisation and other compression before deployment |

## Where it came from

NVIDIA introduced the library in 2016, first under the name GPU Inference Engine, as part of its deep learning SDK, and soon renamed it TensorRT. Early versions focused on convolutional networks for image tasks in data centres and on embedded devices.

Later releases added INT8 quantisation with calibration, an ONNX parser that became the main import path, integration with TensorFlow and PyTorch, support for transformers and dynamic shapes, and FP8 for Hopper-generation GPUs such as the [H100](/tools/nvidia-h100). The Triton server began life as the TensorRT Inference Server. In late 2023 NVIDIA released TensorRT-LLM as open source, targeting language model inference, and newer versions support FP4 on Blackwell-generation GPUs such as the [B200](/tools/nvidia-b200).

## How it works

### Step 1: export the model

The trained model is exported to ONNX, a portable graph format. Alternatively, Torch-TensorRT compiles parts of a PyTorch model directly.

### Step 2: build the engine

The builder parses the graph and applies optimisations:

- **Constant folding and dead-code removal** simplify the graph.
- **Fusion** merges sequences such as convolution, bias and activation, or the parts of attention, into single kernels, which cuts memory traffic.
- **Precision selection** runs layers in FP16, BF16, FP8 or INT8 where allowed, keeping sensitive layers in higher precision.
- **Kernel auto-tuning** times several implementations for each layer on the actual GPU and keeps the fastest.
- **Memory planning** reuses buffers between layers.

```bash
trtexec --onnx=model.onnx --saveEngine=model.plan --fp16
```

This command builds an FP16 engine and reports latency and throughput.

### Step 3: quantise if needed

Lower precision makes inference faster and smaller, but can cost accuracy. INT8 and FP8 need scale factors, found either by calibration on representative data after training or by quantisation-aware training. Tools such as Model Optimizer help apply these methods and check accuracy.

### Step 4: run

The runtime deserialises the engine, allocates input and output buffers and launches inference, often on several CUDA streams in parallel. Engines are specific to a GPU model and TensorRT version, so they are usually built on, or for, the deployment hardware.

### Large language models

TensorRT-LLM provides optimised attention kernels, paged KV cache, in-flight batching, tensor and pipeline parallelism, speculative decoding and quantisation formats suited to LLMs. It offers a Python API and works with Triton and other serving systems.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Engine | The optimised, serialised form of a model for a given GPU |
| Builder | The component that turns a network into an engine |
| Fusion | Merging operations into one GPU kernel |
| Precision | The number format used for computation, such as FP16 or INT8 |
| Calibration | Measuring activation ranges to set INT8 scales after training |
| Quantisation-aware training | Training with simulated low precision to preserve accuracy |
| Dynamic shapes | Inputs whose sizes vary within defined ranges |
| Optimisation profile | The minimum, typical and maximum shapes an engine supports |
| Plug-in | Custom code for an unsupported operation |
| In-flight batching | Adding and removing LLM requests from a batch at every step |
| CUDA stream | A queue of GPU work that can run alongside others |

## A worked example

This scenario is illustrative. A manufacturer runs a visual inspection model on a production line and must process 120 images per second with under 15 milliseconds per image.

1. The PyTorch model runs at 9 milliseconds per image in FP32 on the inspection GPU but only reaches 80 images per second with the required batch size.
2. They export it to ONNX and build an FP16 engine with trtexec. Latency drops to 3.5 milliseconds and throughput rises to about 260 images per second.
3. They try INT8 with calibration on 1,000 representative images. Throughput rises again, but defect recall falls from 98.1 to 96.9 per cent, below their 97.5 per cent requirement.
4. Keeping the first and last layers in FP16 and the rest in INT8 restores recall to 98.0 per cent while keeping throughput well above target.
5. They serve the engine through Triton with dynamic batching, and rebuild engines whenever GPUs or TensorRT versions change.

## Strengths and limitations

**Strengths:**

- Very high inference performance on NVIDIA GPUs.
- Mature support for reduced precision, including FP8 and FP4 on newer hardware.
- Works from data-centre GPUs to embedded Jetson devices.
- Integrates with Triton, ONNX Runtime and PyTorch tooling.
- TensorRT-LLM adds strong LLM-specific optimisations.

**Limitations:**

- NVIDIA GPUs only.
- Engines are tied to a GPU model and library version, so they must be rebuilt.
- Build times can be long for large models.
- Unsupported operations need plug-ins or graph changes.
- Low precision requires careful accuracy checks.

## How it compares with alternatives

| Tool | Hardware | Notes |
|---|---|---|
| TensorRT | NVIDIA GPUs | Peak NVIDIA performance with build step |
| [ONNX Runtime](/tools/onnx-runtime) | CPUs, GPUs, many accelerators | Portable; can use TensorRT as an execution provider |
| [vLLM](/tools/vllm) | NVIDIA, AMD and others | LLM engine, easier set-up |
| [SGLang](/tools/sglang) | NVIDIA, AMD and others | LLM engine with prefix reuse |
| PyTorch with torch.compile | Many | Stays in PyTorch, smaller gains |

## Practical guidance and best practices

1. Measure the original model first, so you can quantify the gain.
2. Start with FP16 or BF16, which usually gives large gains with negligible accuracy loss.
3. Use INT8 or FP8 only with a calibration set that reflects production data, and check accuracy on a held-out set.
4. Keep accuracy-sensitive layers in higher precision when needed.
5. Define optimisation profiles that match real input sizes.
6. Build engines on the same GPU type and TensorRT version as production.
7. Benchmark with realistic batch sizes using trtexec or perf_analyzer.
8. Automate engine builds in CI so upgrades are repeatable.

## Common mistakes

- Copying an engine built for one GPU to a different GPU model.
- Calibrating INT8 with random or unrepresentative data.
- Reporting speed-ups without checking accuracy.
- Setting very wide dynamic shape ranges, which reduces optimisation quality.
- Forgetting that pre- and post-processing may now be the bottleneck.
- Ignoring warnings about layers falling back to slower implementations.

## Frequently asked questions

### Is TensorRT free?

Yes. TensorRT is free to download and use under NVIDIA's licence, and parts of it, including TensorRT-LLM and the open-source components, are on GitHub. It runs only on NVIDIA GPUs.

### Do I need Triton to use TensorRT?

No. Applications can load engines directly with the TensorRT runtime. Triton is useful when you need a server with batching, multiple models and standard APIs.

### What is the difference between TensorRT and TensorRT-LLM?

TensorRT is a general optimiser and runtime for neural networks. TensorRT-LLM builds on it with features specific to language models, such as KV cache management, in-flight batching and LLM parallelism.

### Can ONNX Runtime use TensorRT?

Yes. ONNX Runtime has a TensorRT execution provider, which hands supported parts of a graph to TensorRT and runs the rest with other providers.

### Will lower precision hurt accuracy?

It can. FP16 and BF16 rarely matter for inference. INT8, FP8 and FP4 can cause small or occasionally large losses, so always compare against a validation set before deploying.

## Glossary

- ONNX: an open format for representing machine learning models as graphs.
- Kernel: a function that runs on the GPU.
- FP16 and BF16: 16-bit floating-point formats.
- INT8: 8-bit integer format used for quantised inference.
- FP8 and FP4: 8-bit and 4-bit floating-point formats supported on newer NVIDIA GPUs.
- Jetson: NVIDIA's family of embedded AI computers.
- Torch-TensorRT: a compiler that uses TensorRT from within PyTorch.

## Summary

TensorRT optimises trained neural networks into engines tuned for a specific NVIDIA GPU, using fusion, kernel auto-tuning, memory planning and reduced precision, and TensorRT-LLM extends these ideas to large language models. Introduced in 2016 as the GPU Inference Engine, it remains a standard way to reach peak inference performance on NVIDIA hardware, often behind Triton. The gains are real but come with responsibilities: rebuild engines for each GPU and version, and verify accuracy whenever precision is reduced.