Rust for AI: fast, safe inference, data tooling and where Python still wins

The question behind the question

When someone asks "should we use Rust for AI?", they are usually asking something narrower. They have a model that works in a notebook, and something around it is too slow, too heavy or too fragile. The real question is whether Rust is the right tool for that part.

An AI system is not one program. It is a chain: data comes in, is cleaned and joined, becomes tensors, passes through a model, and the result is served. Each link has a different job and a different cost of failure.

Think of a restaurant. The recipe is invented in a test kitchen where people taste, change one thing and taste again. That is research and training, and it rewards speed of iteration. Dinner service is different: the same dish, hundreds of times a night, on time, with nobody getting ill. That is inference and data plumbing, and it rewards predictability and safety. Python owns the test kitchen. Rust is increasingly found on the line during service.

Why Rust shows up in AI systems

Memory safety without a garbage collector

Rust's compiler tracks who owns each piece of memory and how long every reference lives. Code that could read freed memory, or let two threads write the same data at once, usually does not compile. C and C++ are as fast but leave those checks to you. Go, Java and Python make memory safe with a garbage collector that periodically cleans up in the background.

Rust gets safety without that cleaner. Memory is freed at a known point, when its owner goes out of scope. In a shared office kitchen, one approach is a cleaner who clears the tables now and then, sometimes while you are still eating. Rust's approach is a rule that everyone washes their own cup before leaving.

Predictable latency

No collector means no collector pauses. For a model server this matters most at the tail: average latency is often fine in any language, but the slowest one per cent of requests is what breaks a service-level objective. It does not make the model faster. The matrix maths runs on the same kernels whichever language calls them.

Small binaries and fast start

A Rust service compiles to one native executable, with no interpreter or virtual environment beside it. Hugging Face states this as Candle's core goal: full frameworks such as PyTorch are very large, which makes creating instances on a cluster slow, and Candle aims for lightweight binaries and serverless inference. Text Embeddings Inference lists "small docker images and fast boot times" among its features.

WebAssembly and the edge

Rust compiles to WebAssembly (WASM), which runs in browsers and many edge runtimes. Candle ships WASM examples and in-browser demos of Whisper, LLaMA2, T5 and Segment Anything. Hugging Face tokenizers has a WASM binding on npm, and Burn supports WASM through its WebGPU and pure-Rust CPU backends.

A bridge to Python

PyO3 lets you write a Python module in Rust, and maturin builds it into a normal wheel. This is the quiet reason Rust is already everywhere in AI: the Python packages for tokenizers, safetensors and Polars are all Rust cores with PyO3 bindings. Many Python users run Rust daily without knowing it.

Where Rust fits, and where it does not

Rust is a strong fit when:

• The code runs many times and must not fall over: inference services, tokenisers, request-time features, ingestion workers.
• Latency budgets are tight, especially at the tail, or memory is limited.
• You ship where Python is absent: browsers, mobile and desktop apps, embedded devices, small containers.
• A Python pipeline has one hot loop that NumPy cannot vectorise.
• The work is data infrastructure: query engines, file formats, vector databases.

Rust is a poor fit when:

• You are still deciding what the model is. Research needs notebooks, plotting and every new paper's reference code, and that code is in Python.
• You are training large models. PyTorch, JAX and the distributed and fine-tuning tools around them live in Python. Candle and Burn can train, but the recipes and community knowledge are far thinner.
• The bottleneck is the GPU. If a request spends almost all its time inside a CUDA kernel, rewriting the wrapper changes little.
• The team does not know Rust and the deadline is close. In my experience the learning curve pays back on long-lived services, rarely on a two-week prototype.

Most teams should keep research and training in Python and use Rust selectively for the layers that run in production or at scale.

The ecosystem, organised by job

Figures come from each project's repository and the crates.io index at the time of writing.

Calling model APIs, agents and MCP

A common split is a Python agent calling a Rust MCP server: MCP is the contract, and each side uses the language that suits it. The agentic AI topic covers the agent side.

Serving and inference

TGI is worth studying even as it winds down. It shows a proven shape: Rust at the front door doing networking, queueing and batching, with Python doing the maths behind it.

Vectors and retrieval

Data processing

Arrow is why these fit together. It is an agreed memory layout for columns of data, so Polars, DataFusion, DuckDB, LanceDB and pyarrow can pass data without converting it, like standard shipping containers that every port and lorry can handle without unpacking.

Classic ML and numerics

linfa suits a simple model embedded in a Rust service. It does not match scikit-learn's breadth; for gradient-boosted trees trained in Python, exporting to ONNX and running through ort is often more practical.

Tooling and glue

Three short code examples

These were checked against each project's current README and source.

A first tensor with Candle

This is Candle's own starting example, multiplying two random matrices on the CPU.

use candle_core::{Device, Tensor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let device = Device::Cpu;

    let a = Tensor::randn(0f32, 1., (2, 3), &device)?;
    let b = Tensor::randn(0f32, 1., (3, 4), &device)?;

    let c = a.matmul(&b)?;
    println!("{c}");
    Ok(())
}

With CUDA support, the only change is Device::new_cuda(0)?. Every operation returns a Result, so a shape mismatch is an error you handle, not a crash in a C++ library.

Running an ONNX model with ort

This uses ort = "=2.0.0-rc.15". Pin the exact version while 2.0 is a release candidate; in this candidate, sessions are created from an explicit environment.

use ort::{session::Session, value::Tensor};

fn main() -> ort::Result<()> {
    let env = ort::init().build()?;
    let mut session = Session::builder(&env)?.commit_from_file("model.onnx")?;

    // One row with four features, shape [1, 4].
    let input = Tensor::from_array(([1usize, 4], vec![5.1f32, 3.5, 1.4, 0.2]))?;
    let outputs = session.run(ort::inputs![input])?;

    let (shape, data) = outputs[0].try_extract_tensor::<f32>()?;
    println!("shape {shape:?}, values {data:?}");
    Ok(())
}

Note that run takes &mut self, so one session cannot serve two threads at once; use one session per worker. Also, the default features download a prebuilt ONNX Runtime at build time, which a locked-down build may want to replace.

A PyO3 function that releases the GIL

This module normalises a batch of embedding vectors. py.detach releases Python's global interpreter lock while Rust works, so other Python threads can run.

#[pyo3::pymodule]
mod fastvec {
    use pyo3::prelude::*;

    /// L2-normalise each row of a batch of vectors.
    #[pyfunction]
    fn l2_normalise(py: Python<'_>, rows: Vec<Vec<f32>>) -> Vec<Vec<f32>> {
        py.detach(move || {
            rows.into_iter()
                .map(|mut r| {
                    let n = r.iter().map(|x| x * x).sum::<f32>().sqrt();
                    if n > 0.0 {
                        r.iter_mut().for_each(|x| *x /= n);
                    }
                    r
                })
                .collect()
        })
    }
}

With crate-type = ["cdylib"] and maturin develop --release, Python can import fastvec. Converting nested lists copies every element; for large arrays, rust-numpy works on NumPy memory directly.

Architecture patterns that work

Pattern 1: a Rust inference service

The model is trained in Python, exported to ONNX (or saved as safetensors for Candle) and served by a small Rust binary.

Flow: client → axum or tonic handler (validate, tokenise) → queue → batcher (up to N requests or T milliseconds) → worker threads, one ort session or Candle model each → response, with metrics and traces.

1. Export the model and freeze its input and output names and shapes in a test.
2. Load it once at start-up and fail fast if anything is wrong.
3. Accept requests on tokio, but run inference on dedicated threads.
4. Batch small requests; it is the main lever for GPU throughput.
5. Expose health checks and metrics, and emit OpenTelemetry traces.
6. Ship a small container and scale it as a stateless service.

This is roughly the shape of TEI, so for standard embedding models, use TEI. Write your own when the model or pre-processing is custom. For large LLMs, a dedicated engine will beat a hand-written server. The MLOps and AI infrastructure topic covers deployment.

Pattern 2: a PyO3 extension for one hot path

The pipeline stays in Python. One function, measured as the bottleneck, moves to Rust.

Flow: load → clean → hot loop → train or predict, where the hot loop becomes a call such as fastfeat.compute(...) into a Rust module built with PyO3 and maturin.

1. Profile, and confirm no existing library already does the job.
2. Give the Rust function a narrow interface: arrays in, arrays out.
3. Release the GIL during heavy work; use rayon inside if it splits cleanly.
4. Keep the Python version as a reference and test both give the same output.
5. Build wheels per platform in CI so users only pip install.

This is how tokenizers, safetensors and Polars reached Python users.

Pattern 3: edge and WASM inference

1. Pick a small model, quantised if memory is tight.
2. Build with Candle or Burn for wasm32-unknown-unknown, or ship a native binary with ort for desktop and mobile.
3. Cache the weights; the download is usually slower than the inference.
4. Keep a server fallback for slow devices.

You gain privacy, offline use and no per-request server cost, and pay with size limits and more device testing.

Performance and concurrency, explained simply

Where the time goes. A request has three parts: getting data in and out (networking, parsing, tokenising), the model's maths, and waiting. Rust helps a lot with the first and third, and little with the second when the maths already runs in an optimised kernel. Profile before rewriting.

Ownership makes threads safer. Rust marks which types can move between threads (Send) and which can be shared (Sync), and rejects code that shares mutable data without a lock. It is like a library where a book is lent to one person at a time, or read by many in the reading room but written in by none. Data races mostly become compile errors.

Async for waiting, threads for computing. Tokio handles thousands of mostly idle connections well. A model call is heavy blocking work, and if it runs on an async worker it stalls every request scheduled there. Use dedicated threads, tokio::task::spawn_blocking, or a channel to inference workers.

The GIL and the bridge. Python's global interpreter lock lets one thread run Python bytecode at a time. A PyO3 function that releases it lets Python continue while Rust uses all cores. But crossing the bridge copies data, so call Rust with large batches, not once per row.

Rust, Python and Go for AI work

Python for the test kitchen, Go for the waiting staff and booking system, Rust for the parts of the kitchen that must run at full speed without accidents. Many real systems use all three.

A practical decision guide

1. Is the model still changing every week? Stay in Python.
2. Is there a standard server for your model type? Use it: vLLM for LLMs, TEI for embeddings, llama.cpp or mistral.rs locally.
3. Is one Python function the bottleneck? Write a PyO3 extension for that function only.
4. Must the model run where Python cannot? Use Candle, Burn or ort from Rust.
5. Is the service latency-critical and long-lived? Use ort or Candle behind axum or tonic, and budget time for learning Rust.
6. Is the job data infrastructure? Use the Rust engines already there, Polars, DataFusion, Arrow, Qdrant, LanceDB, from any language.
7. Is the GPU the bottleneck? Change batching, model or quantisation first.

Common mistakes

• Rewriting before measuring, then finding the time was in the model kernel.
• Blocking the async runtime by running inference inside a tokio task.
• One ort session behind one lock for every thread, giving flat throughput.
• Not pinning release candidates. ort 2.0's API has changed between candidates.
• Copying across the Python bridge row by row instead of passing whole arrays.
• Breaking the export contract. Tokeniser, pre-processing and input names must match between Python training and Rust serving; test with real examples on both sides.
• Loading untrusted pickle files instead of safetensors.

Frequently asked questions

Can I train models in Rust?

Yes. Candle supports training, Burn is designed so the same code trains and serves a model, and tch-rs exposes libtorch's training features. What you lose is the surrounding ecosystem: recipes, distributed-training tools, experiment tracking and reference code for new papers. For most teams training stays in Python and Rust handles serving.

Is Rust faster than Python for inference?

For the model's maths, often not by much, because both call the same kernels. The gains come from everything around the model, such as parsing, tokenising, batching, networking and memory use, and from no interpreter or collector overhead. For small models, CPU inference and tight tail-latency targets the difference can be large; for a big LLM on a GPU, the engine's batching and kernels matter far more.

Should I use Candle or ort?

Use ort when you have, or can export, an ONNX model and want ONNX Runtime's optimisations and hardware execution providers. Use Candle for a pure-Rust stack, WASM targets, direct safetensors or GGUF loading, or a model already in candle-transformers. The export path you already have usually decides.

What is PyO3 and why does it matter for AI?

PyO3 lets you write Python modules in Rust, and maturin packages them as ordinary wheels. It lets Rust speed up Python without anyone changing language. Tokenizers, safetensors and Polars all reach Python users this way.

Is there an official Rust SDK for MCP?

Yes. The Model Context Protocol organisation maintains rmcp, built on tokio, with a macros crate for defining tools. At the time of writing it is at 3.9.0 and implements the 2026-07-28 specification while staying compatible with earlier releases. It suits MCP servers that wrap fast Rust code or need a single-binary deployment.

Is Text Generation Inference still a good choice?

Its README says it is in maintenance mode and recommends vLLM and SGLang for new deployments, with llama.cpp or MLX for local use. Its design, a Rust router before a Python model server, remains a useful reference. Text Embeddings Inference is a separate project and is still a strong choice for embeddings and rerankers.

Summary

Rust has found a clear place in AI systems: not in the notebook or the training loop, but in the layers that run every day and must be fast, safe and small. Hugging Face's tokenizers, safetensors, Candle and Text Embeddings Inference, ONNX Runtime through ort, data engines such as Polars, Arrow and DataFusion, vector stores such as Qdrant and LanceDB, and agent tooling such as Rig and the official rmcp SDK show how wide that place has become. Python remains the home of research and training because its ecosystem is unmatched. The practical approach is to keep Python where ideas change quickly, measure where time goes in production, and move only those parts to Rust: as a PyO3 extension, as a lean inference service, or as a model that runs on the edge.