Kubeflow

Overview

Kubeflow is an open-source collection of tools for running machine learning on Kubernetes. It covers the main stages of the work: interactive notebooks for exploration, pipelines for repeatable workflows, operators for distributed training and fine-tuning, automated hyperparameter tuning, a model registry, and integration with serving. Each piece runs as ordinary Kubernetes resources, so it can share a cluster, security model and monitoring with the rest of an organisation's software.

An everyday picture helps. Kubernetes is like the roads, power and water of a town. It keeps things running but does not care what the buildings are for. Kubeflow adds the specialised buildings a research team needs: a lab with benches (notebooks), a production line (pipelines), a heavy workshop for big jobs (distributed training) and a warehouse with a catalogue (the registry). The town's services keep working underneath.

Kubeflow is not a single program to install and forget. It is a platform made of projects that can be used together or separately. That flexibility is its main strength and its main source of complexity.

What it is

Model serving is usually handled by KServe, which began inside Kubeflow and now runs as its own project, with close integration.

The common thread is the custom resource. A user describes what they want, such as "a training job with four workers, each with one GPU", in a YAML or Python definition. A controller in the cluster then creates and supervises the pods needed to make it happen.

Where it came from

Google engineers announced Kubeflow at KubeCon North America in December 2017. The first idea was modest: make it easy to run TensorFlow on Kubernetes, in the way Google ran machine learning internally. Version 0.1 followed in 2018, and the project grew to support many frameworks and to add pipelines, notebooks and tuning.

Kubeflow 1.0 arrived in early 2020, marking several core components as stable. In 2022 Google announced that Kubeflow had applied to join the Cloud Native Computing Foundation, and in July 2023 the CNCF accepted it as an incubating project. In 2025 the training component was redesigned as Kubeflow Trainer, built around a single TrainJob API for different frameworks, with attention to fine-tuning large language models. In August 2026 the CNCF announced that Kubeflow had graduated, its highest maturity level.

How it works

Development in notebooks

A data scientist opens the dashboard and starts a notebook server in their team's namespace, choosing an image and resources such as one GPU. Because the notebook runs in the cluster, it can reach the same storage and services that production jobs use.

Pipelines

A pipeline is a graph of steps, called components, each running in its own container. Steps pass data and artefacts to each other. With the KFP Python SDK, components are ordinary Python functions with a decorator.

from kfp import dsl

@dsl.component
def train(lr: float) -> float:
    return 0.9  # train and return a metric

@dsl.pipeline(name="demo")
def pipeline(lr: float = 0.01):
    train(lr=lr)

The SDK compiles the pipeline into a definition that the Pipelines backend runs. Each run records its inputs, outputs and artefacts, and steps can be cached so unchanged work is not repeated.

Distributed training

For jobs too big for one GPU, Kubeflow Trainer launches several worker pods and wires them together so frameworks such as PyTorch distributed can communicate, often using NCCL on GPU clusters. Reusable runtime definitions let platform teams set images and defaults once, so users only describe the job.

Tuning

Katib runs many trials of a training job with different hyperparameters, using strategies such as random search, grid search or Bayesian optimisation, and reports which settings scored best. Early stopping can end poor trials to save compute.

Multi-tenancy

Profiles give each team a namespace with role-based access, resource quotas and isolated storage. The dashboard shows each user only what they are allowed to see.

Key concepts and terms

A worked example

This scenario is illustrative. A retail analytics team wants to retrain a demand forecasting model every week.

1. They write a pipeline with five components: pull last week's sales from the warehouse, validate the data, build features, train the model and evaluate it against the current production version.
2. The training component runs as a TrainJob with two workers of one GPU each, finishing in about 40 minutes instead of 75 on one GPU.
3. Before the first scheduled run, they launch a Katib experiment with 30 trials over learning rate and tree depth. The best trial improves validation error by 4 per cent, and those settings become pipeline defaults.
4. The pipeline is scheduled every Monday at 02:00. Caching means that when only the training code changes, the data extraction step is reused.
5. If the new model beats production on the holdout set, the pipeline registers it in the Model Registry and updates a KServe deployment; otherwise it stops and alerts the team.

Strengths and limitations

Strengths:

• Open source and vendor-neutral, now a graduated CNCF project.
• Runs on any conformant Kubernetes cluster, in any cloud or on premises.
• Modular: teams can install only Pipelines, or only Trainer.
• Uses Kubernetes-native security, scheduling and GPU management.
• Strong support for distributed training and fine-tuning.

Limitations:

• Installing and upgrading the full platform takes real Kubernetes expertise.
• Many moving parts means many things to monitor and patch.
• Data scientists may find containers and YAML a barrier without good templates.
• Experiment tracking is lighter than dedicated tools, so teams often add one.
• Small teams with a handful of models may not need this much platform.

How it compares with alternatives

Practical guidance and best practices

1. Start with the component you need most, often Pipelines or Trainer, rather than the full platform.
2. Use a managed Kubernetes service if possible, so the team focuses on ML, not cluster upkeep.
3. Build a small set of approved base images with pinned library versions.
4. Keep pipeline components small and single-purpose so caching and reuse work.
5. Set resource requests and quotas per profile to stop one team using every GPU.
6. Pair Kubeflow with an experiment tracker such as MLflow for rich run comparison.
7. Store data and artefacts in object storage, not in pod-local disks.
8. Test upgrades in a staging cluster, because components move at different speeds.

Common mistakes

• Installing every component on day one and then struggling to maintain them.
• Putting heavy data processing inside one giant pipeline step, losing caching and visibility.
• Forgetting GPU node selectors or tolerations, so jobs wait forever for scheduling.
• Running tuning experiments without early stopping, wasting most of the budget on poor trials.
• Giving every user cluster-admin rights instead of using profiles and roles.
• Hard-coding credentials in pipeline code rather than using Kubernetes secrets.

Frequently asked questions

Do I need Kubernetes to use Kubeflow?

Yes. Kubeflow is built on Kubernetes and runs its components as Kubernetes resources. Any conformant cluster works, including managed services from the major clouds and local clusters for testing.

Is Kubeflow only for TensorFlow?

No. It started with TensorFlow, but it now supports PyTorch, JAX, XGBoost and other frameworks, and pipeline steps can run any container.

What is the difference between Kubeflow and Kubeflow Pipelines?

Kubeflow is the whole family of projects. Kubeflow Pipelines is one of them, focused on defining and running workflows. Pipelines can be installed on its own.

How does Kubeflow serve models?

Serving is usually done with KServe, which grew out of Kubeflow. It deploys models as inference services with autoscaling and rollout controls, and integrates with the Kubeflow dashboard and registry.

Is Kubeflow suitable for large language models?

Yes, increasingly. Kubeflow Trainer targets distributed fine-tuning, and KServe has added features for serving generative models. Large jobs still need careful GPU and networking setup underneath.

Glossary

• CNCF: the Cloud Native Computing Foundation, which hosts Kubernetes and Kubeflow.
• Namespace: a Kubernetes boundary that isolates resources for a team or project.
• KFP SDK: the Python library for writing Kubeflow pipelines.
• Runtime: a reusable template that Kubeflow Trainer uses to start jobs.
• Suggestion: the Katib service that proposes the next hyperparameters to try.
• Operator: a controller that manages a specific kind of application on Kubernetes.
• Node selector: a rule that places pods on particular machines, such as GPU nodes.

Summary

Kubeflow turns a Kubernetes cluster into a machine learning platform, adding notebooks, pipelines, distributed training, tuning and a registry as Kubernetes-native resources, with serving through KServe. Started by Google in 2017 and graduated by the CNCF in 2026, it suits organisations that already run Kubernetes and want an open, portable platform. The price of that flexibility is operational effort, so most teams do best by adopting one component at a time and pairing it with a tracker and a managed cluster.

Related

It belongs to MLOps in my stack, under MLOps and AI infrastructure. Others in the same group: