ONNX Runtime

Overview

ONNX Runtime is an open-source engine from Microsoft for running machine learning models stored in the ONNX format. ONNX, the Open Neural Network Exchange, is a common file format that describes a model as a graph of standard operations. A model trained in PyTorch, TensorFlow, scikit-learn or many other tools can be exported to ONNX, and ONNX Runtime can then run it on CPUs, GPUs and specialised accelerators across servers, desktops, phones and web browsers.

An everyday picture: a universal travel adapter. Your laptop has one plug, but sockets differ from country to country. The adapter lets the same plug work everywhere. ONNX is the common plug shape for models, and ONNX Runtime is the adapter set: it lets one exported model use whatever hardware the destination offers, through plug-ins called execution providers.

ONNX Runtime is used both for portability and for speed. It applies graph optimisations before running a model, and it can hand parts of the graph to vendor libraries such as TensorRT or OpenVINO for extra performance, while keeping one application interface.

What it is

Where it came from

Microsoft and Facebook announced the ONNX format in September 2017 so that models could move between frameworks, and many hardware and software companies joined. ONNX later moved under the Linux Foundation's LF AI and Data umbrella, where it is governed as an open standard.

Microsoft open-sourced ONNX Runtime in December 2018 under the MIT licence, after using it in its own products. Over the following years it added many execution providers from hardware vendors, quantisation tools, mobile and web builds, and integrations with Windows. More recently, the ONNX Runtime GenAI library added support for running language models locally and on servers, handling tokenisation, sampling and the KV cache.

How it works

Step 1: export to ONNX

Each framework has an exporter. PyTorch provides torch.onnx.export, and converters exist for TensorFlow, scikit-learn, XGBoost, LightGBM and others. The exported file records operators from a specific opset, a versioned set of operator definitions.

Step 2: create a session

The application creates an InferenceSession with the model and an ordered list of preferred execution providers.

import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession(
    "model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)
name = sess.get_inputs()[0].name
outputs = sess.run(None, {name: np.random.rand(1, 3, 224, 224).astype(np.float32)})

Step 3: optimise the graph

On load, ONNX Runtime applies optimisations at several levels: removing redundant nodes, folding constants, and fusing patterns such as convolution with batch normalisation, or the parts of attention and layer normalisation in transformers. The optimised model can be saved to avoid repeating this work.

Step 4: partition across providers

The runtime asks each execution provider, in priority order, which nodes it can run. Supported subgraphs are assigned to the preferred provider, such as TensorRT on an NVIDIA GPU, and the rest fall back to the next provider, ending with the default CPU provider. This is how one model file can run on very different hardware.

Step 5: execute

The session runs the graph, moving data between providers when needed. Settings control thread counts, memory arenas and whether nodes run in parallel.

Quantisation

Tools in the package convert FP32 models to INT8, either dynamically (weights quantised ahead of time, activations at run time) or statically with calibration data. This shrinks models and speeds up CPU inference, at some risk to accuracy that must be checked.

Key concepts and terms

A worked example

This scenario is illustrative. A software company ships a document classifier inside a desktop application for Windows and macOS, and also runs it on a server.

1. The model is a small transformer fine-tuned in PyTorch. They export it to ONNX with a fixed opset and test that outputs match PyTorch within a tolerance of 1e-4.
2. On the server, ONNX Runtime with the CUDA provider classifies a batch of 32 documents in about 18 milliseconds.
3. For desktops without dedicated GPUs, they apply dynamic INT8 quantisation. The file shrinks from 260 MB to about 70 MB, CPU latency per document falls from 95 to 40 milliseconds, and accuracy drops from 92.4 to 92.0 per cent, which is acceptable.
4. On Windows they enable the DirectML provider, and on macOS the CoreML provider, so machines with capable GPUs or neural engines use them automatically, with CPU fallback otherwise.
5. One model file and one C# and Swift integration now serve every platform.

Strengths and limitations

Strengths:

• One model format and API across many frameworks and platforms.
• Wide hardware support through execution providers.
• Good CPU performance with graph optimisation and quantisation.
• Runs in browsers, on mobile and on edge devices.
• Open source under the MIT licence with strong backing.

Limitations:

• Exporting complex or dynamic models to ONNX can fail or need changes.
• Not every operator is supported by every execution provider, causing slow fallbacks.
• Peak GPU performance may still require vendor-specific tools.
• Debugging differences between frameworks and ONNX outputs takes effort.
• Large language model support is newer and less broad than dedicated LLM engines.

How it compares with alternatives

Practical guidance and best practices

1. Validate exported models by comparing outputs with the original framework on real inputs.
2. Pin the opset version and the ONNX Runtime version in production.
3. List execution providers explicitly and in priority order.
4. Check which nodes fall back to CPU, using verbose logging or profiling.
5. Save optimised models to cut start-up time.
6. Try dynamic quantisation first on CPU, and measure accuracy on a held-out set.
7. Use IO binding on GPUs to avoid needless copies between host and device.
8. Tune thread settings for the target machine rather than relying on defaults.

Common mistakes

• Assuming that because export succeeded, outputs must be identical.
• Exporting with fixed input shapes and then sending other sizes.
• Installing the CPU package and wondering why the GPU is unused.
• Ignoring warnings about unsupported operators on a provider.
• Quantising without measuring accuracy.
• Creating a new session for each request instead of reusing one.

Frequently asked questions

What is the difference between ONNX and ONNX Runtime?

ONNX is the file format and standard for describing models. ONNX Runtime is one engine, from Microsoft, that runs ONNX models. Other engines can also run ONNX files.

Does ONNX Runtime work with GPUs?

Yes. It supports NVIDIA GPUs through CUDA and TensorRT providers, AMD GPUs, DirectML on Windows, and other accelerators through vendor providers. GPU support requires the matching package and drivers.

Can ONNX Runtime run in a web browser?

Yes. ONNX Runtime Web runs models in browsers using WebAssembly on the CPU and WebGPU where available.

Can it run large language models?

Yes, increasingly. The ONNX Runtime GenAI library runs generative models with a full generation loop and KV cache, and is used for local and on-device LLMs. For high-throughput server LLM serving, dedicated engines are more common.

Is ONNX Runtime faster than PyTorch?

Often for inference, especially on CPUs, thanks to graph optimisation and quantisation. The gain depends on the model and hardware, so measure it on your own case.

Glossary

• Exporter: the tool that converts a framework model into ONNX.
• Graph: the network of operators and data flows that makes up a model.
• Constant folding: computing parts of the graph that never change ahead of time.
• WebAssembly: a portable binary format that runs fast code in browsers.
• DirectML: a Windows API for machine learning on many GPUs.
• CoreML: Apple's framework for running models on its devices.
• Calibration data: sample inputs used to choose quantisation scales.

Summary

ONNX Runtime runs models in the open ONNX format on almost any hardware, from servers to phones and browsers, by optimising the graph and assigning parts of it to execution providers for each device. Open-sourced by Microsoft in 2018 and built on the ONNX standard from 2017, it is a common choice for portable, efficient inference and increasingly for local generative models. Its benefits depend on careful export, output validation, explicit provider choice and measured quantisation.

Related

It belongs to Inference and serving in my stack, under MLOps and AI infrastructure. Others in the same group: