# Apache Airflow

Apache Airflow schedules and monitors workflows written as Python DAGs. Learn DAGs, tasks, operators, scheduling, Airflow 3 and its role in data and AI pipelines.

## Overview

Apache Airflow is an open-source platform for writing, scheduling and monitoring workflows as code. Each workflow is a **DAG**, short for directed acyclic graph: a set of tasks connected by dependencies, with no loops. You write the DAG in Python. Airflow decides when each run should start, runs tasks in the right order once their upstream tasks have succeeded, retries failures and shows the state of everything in a web interface.

Airflow is one of the most widely used tools for orchestrating data, analytics and machine learning pipelines. It is good at coordinating many systems, such as databases, warehouses, Spark clusters, Kubernetes jobs and model endpoints, on a schedule or when new data arrives.

A factory production schedule is a helpful picture. Raw materials arrive, parts are machined, assembled, inspected and shipped, and each step starts only after the steps it depends on have finished. A planner sets the schedule, and a board on the wall shows which steps are done, running or stuck. Airflow is that planner and that board for data work. Like a good planner, it does not do the machining itself; it tells the right machine when to start.

## What it is

| Concept | Meaning | Example |
|---|---|---|
| DAG | A workflow of tasks with dependencies and no cycles | `nightly_sales` |
| Task | One unit of work inside a DAG | Run a SQL transformation |
| Operator | A reusable template for a kind of task | SQL, Bash, Kubernetes pod, HTTP operators |
| TaskFlow API | Writing tasks as decorated Python functions | `@task def extract(): ...` |
| Sensor | A task that waits for a condition | Wait for a file in object storage |
| Asset | A named piece of data that tasks produce or consume | `s3://lake/sales/daily` |
| DAG run | One execution of a DAG for a given time or trigger | The run for 4 November |
| Scheduler | Decides which runs and tasks are due | Starts the 02:00 run |
| Executor | Decides where tasks run | Local, Celery workers, Kubernetes |
| Providers | Packages of operators and hooks for external systems | Cloud, database and messaging providers |
| Connections and variables | Stored credentials and configuration | A warehouse connection id |
| XCom | Small values passed between tasks | A file path or row count |

## Where it came from

Maxime Beauchemin created Airflow at Airbnb in 2014 to manage the company's growing number of data pipelines. Airbnb open-sourced it soon after. Airflow entered the Apache Software Foundation's incubator in 2016 and became a top-level Apache project in January 2019.

Airflow 2.0, released in December 2020, brought a highly available scheduler, the TaskFlow API and a stable REST API, followed by regular minor releases up to 2.10 in late 2024. Data-aware scheduling, first called datasets, let DAGs run when upstream data changed rather than only on a clock.

Airflow 3.0 was released in April 2025. It introduced DAG versioning, so a run finishes on the version it started with; a client-server Task Execution Interface with a Python Task SDK, which isolates task code from the core and allows remote execution; an Edge Executor for running tasks outside the main data centre; a rewritten React-based interface; backfills run by the scheduler and started from the UI or API; event-driven scheduling through watchers on message queues; and the renaming of datasets to assets. It also relaxed the rule that each run needs a unique logical date, which suits inference-style workloads.

## How it works

### 1. Define the DAG

You write a Python file that declares a DAG, its schedule and its tasks. Dependencies come from how tasks are wired, either by passing outputs between TaskFlow functions or with the `>>` operator.

```python
from datetime import datetime
from airflow.sdk import dag, task

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def nightly_sales():
    @task(retries=3)
    def extract() -> str:
        return "s3://lake/raw/sales.parquet"

    @task
    def transform(path: str) -> int:
        ...  # start a warehouse job and return the row count

    transform(extract())

nightly_sales()
```

### 2. Parse and schedule

Airflow's DAG processor reads the files and stores each DAG's structure. The scheduler creates DAG runs according to the schedule, an asset update or a manual trigger, and marks tasks as ready when their upstream tasks have succeeded.

### 3. Execute

The executor sends ready tasks to workers: local processes, a Celery worker pool or individual Kubernetes pods. In Airflow 3 tasks talk to an API server through the Task SDK rather than directly to the metadata database.

### 4. Track, retry and alert

Each task instance has a state, such as queued, running, success, failed or up for retry. Logs are stored per attempt. Failed tasks retry according to their settings, and callbacks or notifiers can alert people. Downstream tasks wait.

### 5. Re-run history

When logic changes or data is corrected, backfills re-run past intervals, and individual tasks can be cleared and re-run from the interface.

## Key concepts and terms

| Term | Meaning |
|---|---|
| DAG | Directed acyclic graph of tasks and dependencies |
| Task instance | One run of one task for one DAG run |
| Operator | A class that defines a type of task |
| Sensor | A task that waits for a condition before succeeding |
| Asset | A data object used for data-aware scheduling |
| Logical date | The date or interval a DAG run represents |
| Catchup | Creating runs for past intervals since the start date |
| Backfill | Deliberately re-running historical intervals |
| Executor | The component that decides how and where tasks run |
| Provider | A package of integrations for an external system |
| XCom | A mechanism for passing small values between tasks |
| Idempotent task | A task that gives the same result when run again |

## A worked example

This is an illustrative scenario. A retailer with 120 stores runs a nightly DAG at 02:00.

1. **Extract.** A task copies the previous day's sales, about 3 million rows, from the store database into object storage. It has three retries with a ten-minute delay.
2. **Load and transform.** A task starts transformation models in the warehouse, for example with [dbt](/tools/dbt), and records the row count.
3. **Check.** A data-quality task fails the run if the row count is below 80 percent of the seven-day average, which usually means a store did not upload.
4. **Drift.** A task compares this week's feature distributions with the training data. If drift exceeds a threshold, a branch triggers a retraining job on [Kubernetes](/tools/kubernetes); otherwise it skips.
5. **Score.** A task calls the forecasting model to predict next week's demand per store and product, and writes results to the warehouse, updating an asset.
6. **Summarise.** A task sends the largest changes to a language model, which writes a short note for the morning report.
7. **Publish.** A separate dashboard DAG is scheduled on the forecast asset, so it runs as soon as the asset updates.

One night the store database is down for twenty minutes. The extract task fails twice, succeeds on the third try, and every downstream task simply starts later. On another night the quality check fails, and the on-call engineer is alerted before anyone sees wrong numbers.

## Strengths and limitations

**Strengths:**

- Pipelines as Python code, which can be reviewed, tested and versioned.
- A very large library of providers and operators.
- Mature scheduling, retries, backfills and monitoring.
- Data-aware and event-driven scheduling since recent versions.
- Large community and managed offerings on major clouds.

**Limitations:**

- Not designed for low-latency streaming or per-event processing.
- Scheduling overhead makes it a poor fit for many tiny, second-long tasks.
- Large DAG files and heavy top-level code slow parsing.
- Passing large data through XCom is a common performance problem.
- Long-lived, interactive processes that wait for people suit other tools better.

## How it compares with alternatives

| Option | Choose it when | Trade-off |
|---|---|---|
| Airflow | Batch data and ML pipelines across many systems | Batch-oriented, scheduling overhead |
| [Dagster](/tools/dagster) | You want an asset-centric model with strong typing and testing | Smaller ecosystem |
| [Prefect](/tools/prefect) | You want Python-native flows with minimal boilerplate | Different operating model |
| [Temporal](/tools/temporal) | You need durable, long-running application workflows | Not a data scheduler |
| Streaming engines such as [Apache Flink](/tools/apache-flink) | You need continuous, low-latency processing | Different programming model |

## Practical guidance and best practices

1. Make every task idempotent, so retries and backfills give the same result.
2. Let Airflow orchestrate and push heavy computation to warehouses, Spark or Kubernetes.
3. Keep top-level DAG code light; no database calls or large imports at parse time.
4. Pass references, such as file paths, between tasks, not large data.
5. Use assets to trigger downstream DAGs on data updates instead of guessing times.
6. Add data-quality checks that fail fast before bad data spreads.
7. Store credentials in connections or a secrets backend, never in DAG code.
8. Test DAGs in CI, including import checks, and deploy through version control.
9. Set alerts and expected completion times on critical pipelines.

## Common mistakes

- Doing heavy processing inside the Airflow worker instead of an external engine.
- Writing tasks that append duplicate rows when retried.
- Leaving `catchup` on by accident and creating hundreds of runs for past dates.
- Using XCom to pass whole data frames between tasks.
- Hiding many unrelated jobs in one giant DAG.
- Expecting Airflow to process streaming events in real time.

## Frequently asked questions

### What is Apache Airflow?

It is an open-source platform for writing, scheduling and monitoring workflows as Python code. Workflows are DAGs of tasks, and Airflow runs them in dependency order on schedules, data updates or manual triggers.

### What is a DAG in Airflow?

A directed acyclic graph: a set of tasks with dependencies and no cycles. It defines what runs and in which order. Each execution of the DAG is a DAG run.

### What changed in Airflow 3?

Airflow 3.0, released in April 2025, added DAG versioning, a Task Execution Interface with a Task SDK for isolated and remote task execution, an Edge Executor, a new React interface, scheduler-managed backfills, event-driven scheduling and the renaming of datasets to assets.

### Can Airflow run AI and ML workloads?

Yes. It commonly orchestrates data preparation, training, evaluation, batch scoring and AI enrichment steps. Usually it triggers external compute, such as Kubernetes jobs or managed training services, rather than running models inside its own workers.

### Is Airflow suitable for streaming?

Not for low-latency stream processing. It can react to events and start runs when messages arrive, but continuous per-event processing belongs in a streaming engine.

### Should AI agents be built in Airflow?

Airflow can schedule agent jobs, such as a nightly research or summarisation task. Interactive agents that wait for people or run for days are better served by durable execution engines or agent frameworks.

## Glossary

- DAG: a graph of tasks with dependencies and no cycles.
- Task: a single unit of work.
- Operator: a template for a type of task.
- Sensor: a task that waits for a condition.
- Asset: a data object that can trigger DAGs.
- Scheduler: the component that creates runs and queues tasks.
- Executor: the component that runs tasks on workers.
- Backfill: re-running past intervals.
- Provider: a package of integrations.
- XCom: small values shared between tasks.

## Summary

Apache Airflow orchestrates batch data and AI pipelines as Python DAGs, with scheduling, data-aware triggers, retries, backfills and monitoring. Created at Airbnb in 2014 and an Apache top-level project since 2019, it reached version 3 in April 2025 with DAG versioning, a Task SDK and event-driven scheduling. It excels at coordinating many systems, not at doing heavy computation or real-time streaming itself. Keep tasks idempotent, push heavy work outward, pass references rather than data and treat DAGs like production code.