MLflow

Overview

MLflow is an open-source platform for managing the life of a machine learning model, from the first experiment to production and beyond. It records what each training run did, packages the resulting model in a standard format, keeps versions in a registry, and, in its recent releases, traces and evaluates applications built on large language models and agents.

Think of a careful chef developing a new recipe. Each attempt is written down: the oven temperature, the amount of salt, the baking time and how the dish tasted. When a version finally works, the chef can cook it again exactly, and can explain to anyone why it is better than last week's attempt. MLflow is that recipe notebook for data science teams, except that it writes itself while the code runs.

The value is less about any single feature and more about discipline. Without a tracking system, teams lose track of which data, code and settings produced which model. With MLflow, that history is captured by default, so models can be compared, reproduced, audited and promoted with confidence.

What it is

MLflow is a set of loosely coupled components. A team can adopt one, such as tracking, without using the rest.

The tracking server needs two kinds of storage. A backend store, usually a relational database, holds metadata such as parameters and metrics. An artefact store, usually object storage such as Amazon S3, holds large files such as model weights, plots and datasets.

Where it came from

Databricks announced MLflow in June 2018 as an open-source project. The aim was to give teams a lifecycle tool that did not lock them into one framework or one cloud. Early versions centred on three ideas: tracking, projects (a format for packaging reproducible code) and models. The Model Registry was added in 2019.

In June 2020 MLflow joined the Linux Foundation, which gave it vendor-neutral governance while Databricks continued to contribute heavily and offer a managed version on its platform. Through the 2.x series the project added LLM-oriented features, and the registry moved from fixed lifecycle stages towards flexible aliases and tags.

MLflow 3 was released in June 2025. It reorganised the project around generative AI as well as classic machine learning. It introduced a first-class LoggedModel entity, so a model or agent version can be tracked on its own rather than only as an output of a run, and it expanded tracing, evaluation and prompt management for LLM applications.

How it works

Logging a run

A training script opens a run, logs what it used and what it measured, and closes the run. Each run belongs to an experiment, which is simply a named group of runs.

import mlflow

mlflow.set_experiment("credit-risk")
with mlflow.start_run():
    mlflow.log_param("max_depth", 6)
    mlflow.log_metric("val_auc", 0.871)
    mlflow.set_tag("data_version", "2026-10-01")

For common libraries, mlflow.autolog() captures parameters, metrics and the trained model automatically, so most of this code is not needed.

Packaging a model

A logged model is saved as a directory containing the weights, a description file called MLmodel, the environment needed to run it, and optionally a signature describing expected inputs and outputs. The flavour system means a scikit-learn model, a PyTorch model or a custom Python function can all be loaded through one generic interface.

Registering and promoting

When a model is good enough, it is registered under a name, which creates version 1, then version 2, and so on. Aliases such as champion or challenger point at specific versions. A serving system asks for models:/credit-risk@champion instead of a hard-coded version, so promoting a model is a matter of moving the alias.

Tracing and evaluating LLM applications

For LLM applications, MLflow records each request as a trace made of spans: the prompt, the retrieval step, each tool call and the final answer, with timings and token counts. Evaluation runs a dataset through the application and scores the outputs, using rule-based checks or an LLM as a judge. The same experiment view then shows quality alongside cost and latency.

Key concepts and terms

A worked example

This scenario is illustrative. A lending team is building a model to predict loan default.

1. Over two weeks the team runs 180 experiments across gradient boosting and logistic regression, varying features and hyperparameters. Autologging records every run.
2. In the comparison view they sort by validation AUC and filter to runs where a fairness metric, the gap in approval rates between groups, is below 0.03. Twelve runs qualify.
3. The best qualifying run has an AUC of 0.874. They open it, see it used data snapshot 2026-10-01 and commit a41c9e2, and re-run it to confirm the result reproduces within 0.002.
4. They register it as version 7 of loan-default and assign the challenger alias. A shadow deployment scores live traffic for a week next to the current champion, version 5.
5. The challenger performs better on recent data, so the team moves the champion alias to version 7. Rolling back would simply mean moving the alias back.
6. When an auditor later asks why version 7 was approved, the team exports the run's parameters, data version, metrics and review notes from MLflow.

Strengths and limitations

Strengths:

• Open source under the Apache 2.0 licence, with neutral governance.
• Works with almost any framework, language binding or cloud.
• Components can be adopted one at a time.
• Autologging gives value with very little code.
• Large ecosystem of integrations, including managed versions from several cloud providers.

Limitations:

• Self-hosting means running a tracking server, a database and object storage, plus authentication.
• Access control and governance in the open-source server are basic compared with managed offerings.
• The UI can feel slow with very large numbers of runs or metrics.
• Some newer generative AI features have appeared first in the managed Databricks product.
• It does not orchestrate pipelines or serve models at scale by itself; it integrates with tools that do.

How it compares with alternatives

Practical guidance and best practices

1. Log the data version or snapshot identifier with every run, not only the code commit.
2. Turn on autologging first, then add manual logging only for what it misses.
3. Use a shared tracking server from the start, so runs do not live on individual laptops.
4. Store artefacts in durable object storage, and keep the backend database backed up.
5. Promote models with aliases rather than hard-coding version numbers in serving code.
6. Record model signatures, so serving systems can reject malformed inputs early.
7. Name experiments by project and purpose, and use tags for owner, dataset and ticket.
8. For LLM applications, enable tracing in development so evaluation and debugging use the same records.

Common mistakes

• Logging metrics but not the data version, which makes results impossible to reproduce.
• Running the tracking server with a local file store and losing history when the machine changes.
• Pointing production at "latest version" instead of a reviewed alias.
• Logging huge artefacts on every step, which fills storage and slows the UI.
• Treating the registry as approval in itself, without a review or evaluation step.
• Leaving the tracking server open without authentication on a shared network.

Frequently asked questions

Is MLflow free to use?

Yes. MLflow is open source under the Apache 2.0 licence and can be self-hosted at no licence cost. Several vendors, including Databricks and major cloud providers, also offer managed MLflow, which is paid as part of their platforms.

Do I need Databricks to use MLflow?

No. MLflow runs anywhere Python runs, with a tracking server on a virtual machine, a container or Kubernetes. Databricks offers a managed and more tightly integrated version, but the open-source project stands on its own.

What is the difference between a run and a registered model?

A run records one training or evaluation attempt with everything it logged. A registered model is a named, versioned entry in the registry that points at a model produced by a run. Many runs are exploratory; only a few produce models worth registering.

Can MLflow serve models?

MLflow can start a simple local REST server for a model and can build container images for deployment. For high-traffic production serving, teams usually load MLflow models into a serving platform such as KServe, BentoML or a managed endpoint.

How does MLflow help with LLM applications?

It records traces of each request, versions prompts, and evaluates outputs against datasets with automated scorers or LLM judges. This gives LLM applications the same reproducibility and comparison that tracking gives classic models.

Should I use stages or aliases in the registry?

Aliases. The older fixed stages, such as Staging and Production, were deprecated in favour of aliases and tags, which are more flexible and allow several named pointers at once.

Glossary

• Backend store: the database holding run metadata such as parameters and metrics.
• Artefact store: object storage holding large files produced by runs.
• MLmodel file: the descriptor saved with each model, listing its flavours and environment.
• LoggedModel: an MLflow 3 entity that tracks a model or agent version independently of a single run.
• Span: one timed step inside a trace, such as a retrieval or tool call.
• Champion and challenger: common alias names for the current production model and its candidate replacement.
• Model URI: an address such as models:/name@alias used to load a model.
• Lineage: the recorded link between a model, its data, code and parameters.

Summary

MLflow gives machine learning teams a shared memory: every run's settings, data, results and outputs are recorded, good models are packaged and versioned in a registry, and promotion happens by moving aliases rather than editing code. Created by Databricks in 2018 and governed under the Linux Foundation since 2020, it has grown with MLflow 3 into a tool for tracing and evaluating LLM and agent applications too. It is not a pipeline orchestrator or a large-scale serving system, but it fits alongside them, and it is often the simplest first step towards reproducible, auditable machine learning.

Related

It belongs to MLOps in my stack, under MLOps and AI infrastructure. Others in the same group: