# Kubernetes

Kubernetes runs and manages containers across clusters: scheduling, scaling, self-healing and rolling updates. Learn pods, deployments, services and its role in AI.

## Overview

Kubernetes is open-source software for running containerised applications across a group of machines, called a cluster. You describe what you want, for example "three copies of this web service, each with one GPU, reachable at this address", and Kubernetes makes it happen: it chooses machines, starts containers, restarts them if they fail, spreads traffic among them, scales them up and down and rolls out new versions without downtime. It is often abbreviated to K8s, for the eight letters between the K and the s.

An everyday picture: an orchestra conductor. The musicians are containers, each playing their part. The score says what should be playing. The conductor does not play an instrument, but watches constantly, brings in replacements when someone stops, adjusts sections when the piece gets louder, and keeps everyone in time. Kubernetes conducts containers according to the score you write, which is why it is called a container orchestrator.

Kubernetes has become the standard platform for cloud-native software, and increasingly for AI. Training jobs, inference services, data pipelines and the tools around them, such as [Kubeflow](/tools/kubeflow), [KServe](/tools/kserve) and Ray, commonly run on Kubernetes, which schedules scarce GPUs among many teams.

## What it is

| Part | Purpose |
|---|---|
| Control plane | The components that manage the cluster: API server, scheduler, controllers and etcd |
| API server | The front door; every change goes through it |
| etcd | The key-value store holding the cluster's desired and current state |
| Scheduler | Chooses a node for each new pod based on resources and rules |
| Controller manager | Runs control loops that make reality match the desired state |
| Nodes | Worker machines that run pods |
| kubelet | The agent on each node that starts and monitors containers |
| Container runtime | Software such as containerd that runs containers |
| kube-proxy and CNI | Networking components that connect pods and services |
| kubectl | The command-line tool for talking to the API server |

On top of these sit the objects users work with: pods, deployments, services and many more, described below.

## Where it came from

Kubernetes grew out of more than a decade of experience at Google running containers at enormous scale with internal systems called Borg and later Omega. Google announced Kubernetes as an open-source project in June 2014, with engineers including Joe Beda, Brendan Burns and Craig McLuckie among its founders. The name comes from the Greek word for a helmsman or pilot.

Version 1.0 was released in July 2015, and Google donated the project to the newly formed Cloud Native Computing Foundation, part of the Linux Foundation. In March 2018 Kubernetes became the CNCF's first graduated project. All major cloud providers now offer managed Kubernetes, and new minor versions are released roughly three times a year. Recent releases have focused heavily on AI workloads; for example, Dynamic Resource Allocation, a more flexible way to request GPUs and other devices, became stable in version 1.34 in 2025.

## How it works

### Desired state and control loops

The central idea is declarative configuration. You submit an object, usually written in YAML, describing the desired state. Controllers watch the API server, compare desired state with actual state, and act to close the gap. If a node dies and three replicas become two, the controller notices and creates a third elsewhere. This loop runs continuously, which is what makes Kubernetes self-healing.

### Pods

A pod is the smallest deployable unit: one or more containers that share a network address and storage volumes and are scheduled together. Most pods run one main container, sometimes with helper "sidecar" containers.

### Workload objects

- **Deployment** manages identical, stateless pods and performs rolling updates.
- **StatefulSet** manages pods that need stable names and storage, such as databases.
- **DaemonSet** runs one pod on every node, for agents such as log collectors or GPU drivers.
- **Job** and **CronJob** run tasks to completion, once or on a schedule.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: embedder
spec:
  replicas: 2
  selector: { matchLabels: { app: embedder } }
  template:
    metadata: { labels: { app: embedder } }
    spec:
      containers:
        - name: server
          image: registry.example.com/embedder:1.4
          resources:
            limits: { nvidia.com/gpu: 1 }
```

### Networking

A **Service** gives a stable address to a changing set of pods and load-balances among them. An **Ingress** or the newer **Gateway API** routes external HTTP traffic to services. Every pod can reach every other pod unless network policies restrict it.

### Scheduling and resources

Each container declares resource requests, the amount guaranteed, and limits, the maximum allowed. The scheduler places pods only where requests fit. Labels, node selectors, affinity rules, taints and tolerations steer pods to suitable machines, such as GPU nodes.

### GPUs

GPUs are exposed by device plug-ins, usually installed by a vendor operator such as the NVIDIA GPU Operator, which also manages drivers. Pods request GPUs as whole units, or through features such as Multi-Instance GPU, time-slicing or Dynamic Resource Allocation for finer control.

### Scaling

The Horizontal Pod Autoscaler adds or removes pods based on CPU, memory or custom metrics. A cluster autoscaler, or tools such as Karpenter, adds or removes nodes when pods cannot be scheduled or nodes sit idle.

### Configuration and extension

ConfigMaps and Secrets inject configuration. Persistent volumes provide durable storage. Custom resource definitions and operators extend Kubernetes with new object types, which is how projects such as KServe and Kubeflow add their own resources.

## Key concepts and terms

| Term | Meaning |
|---|---|
| Cluster | A set of nodes managed by one control plane |
| Node | A worker machine, physical or virtual |
| Pod | One or more containers scheduled together |
| Deployment | Manages replicas of stateless pods and their updates |
| Service | A stable network address for a set of pods |
| Namespace | A virtual partition of a cluster for teams or projects |
| Label | A key-value tag used to select objects |
| Request and limit | Guaranteed and maximum resources for a container |
| Taint and toleration | Rules that keep pods off nodes unless they opt in |
| Operator | A controller that manages a complex application |
| Custom resource | A user-defined object type added to the API |
| Horizontal Pod Autoscaler | Scales pod counts based on metrics |

## A worked example

This scenario is illustrative. A company runs a text embedding service and nightly batch scoring on a managed Kubernetes cluster.

1. The cluster has a CPU node pool and a GPU node pool. GPU nodes carry a taint so only pods that tolerate it, and request GPUs, land there.
2. The embedding service is a Deployment with two replicas, each requesting one GPU, behind a Service. A Horizontal Pod Autoscaler scales it from 2 to 8 replicas based on requests per second exported to [Prometheus](/tools/prometheus).
3. During a product launch, traffic quadruples. The autoscaler asks for six more pods; four cannot be scheduled, so the cluster autoscaler adds two GPU nodes within a few minutes.
4. A new model version is released by changing the image tag. The Deployment replaces pods one at a time, waiting for each to pass its readiness check, so the service never goes down.
5. Each night a CronJob starts a batch scoring Job using spare GPU capacity. Overnight, unused nodes are removed to save cost.
6. Configuration is kept in git and applied with [Helm](/tools/helm) charts through [Argo CD](/tools/argocd), so every change is reviewed and reversible.

## Strengths and limitations

**Strengths:**

- Self-healing, declarative management of containers at scale.
- Portable across clouds and on-premises data centres.
- Rich ecosystem of tools, operators and managed services.
- Rolling updates, autoscaling and service discovery built in.
- Strong support for sharing GPUs and other accelerators among teams.

**Limitations:**

- Steep learning curve with many concepts and objects.
- Operating clusters, especially self-managed, takes real expertise.
- Easy to misconfigure security, networking and resource limits.
- Overkill for a few simple services.
- Stateful and very large-scale training workloads need extra tooling.

## How it compares with alternatives

| Platform | Approach | Best fit |
|---|---|---|
| Kubernetes | Open-source container orchestration | Many services, portability, shared GPU platforms |
| [Docker](/tools/docker) Compose | Multi-container apps on one machine | Development and small deployments |
| HashiCorp Nomad | Simpler scheduler for containers and other workloads | Teams wanting less complexity |
| Slurm | HPC batch scheduler | Large training clusters, research computing |
| Managed container services | Cloud-run containers without managing clusters | Simple services with little operations staff |

## Practical guidance and best practices

1. Use a managed Kubernetes service unless you have strong reasons and skills to run your own.
2. Set resource requests and limits for every container.
3. Add readiness and liveness probes so traffic and restarts behave correctly.
4. Separate GPU nodes with taints and dedicated node pools.
5. Use namespaces, role-based access control and network policies to isolate teams.
6. Store manifests in git and deploy through CI/CD or GitOps tools.
7. Keep secrets in a secret manager such as [HashiCorp Vault](/tools/hashicorp-vault) or the cloud provider's service.
8. Upgrade clusters regularly, since each minor version is supported for a limited time.

## Common mistakes

- Running containers without resource requests, causing noisy neighbours and evictions.
- Using the `latest` image tag, which makes rollouts unpredictable.
- Granting cluster-admin rights widely.
- Storing secrets in plain manifests in git.
- Leaving GPU nodes running idle because nothing scales them down.
- Ignoring readiness probes, so pods receive traffic before models load.

## Frequently asked questions

### What is the difference between Docker and Kubernetes?

[Docker](/tools/docker) builds and runs containers, usually on one machine. Kubernetes runs and manages containers across many machines, handling scheduling, scaling, networking and recovery. They work together: images built with Docker run on Kubernetes.

### Do I need Kubernetes for machine learning?

Not always. A few models can run on managed endpoints or simple servers. Kubernetes becomes valuable when many teams share GPUs, many services must be operated consistently, or portability across clouds matters.

### How does Kubernetes handle GPUs?

Through device plug-ins and operators that expose GPUs as schedulable resources. Pods request GPUs in their resource limits, and the scheduler places them on nodes with free GPUs. Newer features allow sharing and finer-grained requests.

### Is Kubernetes free?

The software is open source under the Apache 2.0 licence. You pay for the machines it runs on, and managed services usually charge a fee for the control plane.

### What is a Kubernetes operator?

An operator is a controller plus custom resources that encode how to run a complex application, such as a database or a training job. Users create a high-level object, and the operator manages the details.

## Glossary

- Container: a packaged application with its dependencies, run in isolation.
- Manifest: a YAML or JSON file describing Kubernetes objects.
- Control loop: a process that repeatedly moves actual state towards desired state.
- Rolling update: replacing pods gradually with a new version.
- Readiness probe: a check that a pod can receive traffic.
- Liveness probe: a check that restarts a pod if it is stuck.
- Helm: a package manager for Kubernetes applications.
- GitOps: managing infrastructure by applying configuration stored in git.

## Summary

Kubernetes is an open-source container orchestrator that keeps a cluster in the state you declare, scheduling pods, restarting failures, balancing traffic, scaling and rolling out updates. Born from Google's experience with Borg, open-sourced in 2014 and the CNCF's first graduated project in 2018, it is now the common foundation for cloud-native and AI platforms, including GPU sharing for training and inference. It brings real power and real complexity, so most teams do best with managed clusters, strict resource and security settings, and configuration kept in version control.