Dagster

Overview

Dagster is an open-source orchestrator for data pipelines, written in Python. An orchestrator decides what runs, when, in what order and what happens when something fails. Most orchestrators describe pipelines as tasks: run this script, then that one. Dagster's main abstraction is the asset: a table, file, model or report that a pipeline produces. You declare each asset in Python along with the assets it depends on, and Dagster works out the order, runs the code, records the result and shows the lineage.

Thinking in assets changes everyday questions. Instead of "did the 3 a.m. job succeed?", you ask "is the daily_revenue table up to date, and what does it depend on?" Dagster's web interface shows a graph of assets, when each was last materialised, which partitions exist, whether quality checks passed and which upstream change made something stale. It also supports traditional task-style jobs, schedules, event-driven sensors, partitions and backfills, and it integrates with tools such as dbt, Spark, Snowflake and many cloud services.

An analogy: a task-based orchestrator is a kitchen timetable that says "at 5 p.m. chop onions, at 5.10 fry them". Dagster is a menu with ingredient lists. It knows that the soup needs stock, the stock needs bones, and the bones were delivered this morning. If the stock is already fresh, it does not make it again; if the bones change, it knows the soup is now out of date.

What it is

Where it came from

Dagster was started by Nick Schrock, formerly an engineer at Facebook and a co-creator of GraphQL, who founded the company Elementl in 2018 to build it. The project was released publicly as open source in 2019 under the Apache License 2.0. Elementl later renamed itself Dagster Labs and offers a hosted service, now called Dagster+.

Dagster 1.0 was announced on 5 August 2022, declaring the core abstractions stable. Around that time software-defined assets became the centre of the framework. Later releases added declarative automation conditions, asset checks, Dagster Pipes for running code in external environments and a components system with a dg command-line tool for structuring projects. In July 2026 Prefect, the company behind the Prefect orchestrator, announced that it was acquiring Dagster Labs. The announcement said the Dagster brand and open-source project would continue and be actively developed.

How it works

Defining assets

You write a Python function and decorate it with @asset. The function's parameters name the upstream assets it depends on. Dagster builds a dependency graph from these declarations. The function's return value is handed to an I/O manager, which stores it, for example as a Parquet file or a warehouse table. Assets can also be declared from external tools: a dbt project becomes a set of assets, one per model, with dependencies read from dbt's manifest.

import dagster as dg

@dg.asset
def raw_orders() -> list[dict]:
    return [{"id": 1, "amount": 25.0}, {"id": 2, "amount": 40.0}]

@dg.asset
def daily_revenue(raw_orders: list[dict]) -> float:
    return sum(o["amount"] for o in raw_orders)

defs = dg.Definitions(assets=[raw_orders, daily_revenue])

Running and automating

1. Dagster loads your definitions from one or more code locations.
2. A run is launched manually, by a schedule, by a sensor or by an automation condition, for example "materialise when upstream data has been updated".
3. The run launcher starts the run in a process, container or Kubernetes pod, depending on your deployment.
4. Each step runs in dependency order. Resources provide connections and credentials. Logs and metadata, such as row counts, are recorded.
5. Asset checks run after materialisation and record pass or fail results.
6. The UI updates the asset catalogue with the new materialisation, its metadata and lineage.

Partitions and backfills

An asset can be partitioned, for example by day. Dagster then tracks which partitions are materialised. A backfill materialises a range of partitions, such as the past 90 days, in parallel and shows progress per partition.

Deployment

The main services are the web server for the UI and API, a daemon that runs schedules, sensors and automation, and the run workers. Metadata is stored in a database such as PostgreSQL. You can deploy on a single machine, on Kubernetes or with the hosted Dagster+.

Key concepts and terms

A worked example

This scenario is illustrative. A small analytics team maintains 60 dbt models in a warehouse, two Python ingestion jobs and a weekly churn model. Their previous setup was a set of cron jobs, and failures were often noticed only when a dashboard looked wrong.

1. They load the dbt project into Dagster, which creates 60 assets with dependencies taken from dbt.
2. They write the two ingestion jobs as Python assets, partitioned by day, that land raw data in the warehouse. The first dbt models depend on them.
3. They define the churn model as an asset that depends on two dbt models, and its predictions as a downstream asset.
4. Asset checks verify that orders has no duplicate IDs and that yesterday's row count is within 30 per cent of the previous week's average.
5. Ingestion runs on a daily schedule. The dbt models use automation conditions to update once their upstream partitions land. The churn model runs weekly.
6. One morning an ingestion source fails. The UI shows the failed partition and marks downstream assets as stale. After the source recovers, the team re-runs that partition, and the dependent assets update automatically.
7. When they change a column in one dbt model, the lineage graph shows exactly which downstream tables and the churn model are affected.

Strengths and limitations

Strengths:

• Asset-centric model gives clear lineage, freshness and impact analysis.
• Strong local development and testing; definitions are ordinary Python.
• First-class partitions, backfills and data quality checks.
• Good integrations, especially with dbt and modern warehouses.
• A rich web UI for operations and an asset catalogue.

Limitations:

• The asset model and its many concepts take time to learn.
• Moving from task-based tools requires rethinking pipelines, not just porting code.
• Self-hosting at scale needs care with the daemon, run launchers and metadata database.
• A smaller community than Airflow, with fewer ready-made operators.
• Very dynamic, task-heavy workflows can feel less natural than in task-first tools.

How it compares with alternatives

Practical guidance and best practices

1. Model the things you care about, tables, files and models, as assets, and keep each asset's function focused.
2. Use resources for all external connections so code is testable and environments can be swapped.
3. Partition large time-based assets so failures and backfills affect only the slices involved.
4. Add asset checks for keys, row counts and freshness on important assets.
5. Write unit tests that call asset functions directly with small inputs and mock resources.
6. Split large projects into code locations by team or domain so deployments stay independent.
7. Prefer declarative automation or sensors over many overlapping cron schedules.
8. Record useful metadata, such as row counts and previews, on each materialisation.

Common mistakes

• Wrapping an entire legacy script in one giant asset and losing the lineage benefits.
• Hard-coding credentials and connections instead of using resources.
• Creating very fine-grained partitions, such as per hour per customer, that overwhelm the UI and metadata store.
• Ignoring asset checks and relying on dashboards to reveal bad data.
• Treating Dagster as a compute engine; heavy processing should run in a warehouse, Spark or similar.
• Running schedules and sensors without the daemon deployed.

Frequently asked questions

How is Dagster different from Airflow?

Airflow is organised around tasks in DAGs and focuses on running them on time. Dagster is organised around the data assets those tasks produce, so lineage, freshness, partitions and quality checks are built into the model. Airflow has a larger ecosystem and long history; Dagster offers a more data-aware development and testing experience.

Do I have to use assets?

No. Dagster also supports ops, graphs and jobs for task-style work. Assets are recommended for pipelines that produce data, because they unlock most of Dagster's lineage and automation features.

Does Dagster work with dbt?

Yes. The dbt integration turns each dbt model into a Dagster asset, keeps dbt's dependencies, and lets you schedule, partition and observe dbt models alongside Python assets.

Is Dagster open source?

Yes. The core framework is open source under the Apache License 2.0. Dagster Labs also offers a hosted commercial service, Dagster+.

What does the Prefect acquisition mean for Dagster users?

In July 2026 Prefect announced it was acquiring Dagster Labs. The announcement said Dagster would keep its brand, that customers' service and contracts were unaffected and that the open-source project would continue to be actively developed. Check the project's own channels for the latest roadmap.

Can Dagster orchestrate machine learning pipelines?

Yes. Training data, features, models and evaluation results can all be assets, with partitions for time-based data and checks for model quality. Heavy training typically runs on external compute, triggered and tracked by Dagster.

Glossary

• Backfill: running an asset for a range of past partitions.
• Daemon: the Dagster process that evaluates schedules, sensors and automation.
• Dagster+: the hosted commercial version of Dagster.
• Dagster Pipes: a protocol for running code in external environments and reporting back to Dagster.
• Freshness: how recently an asset was updated compared with expectations.
• Op: a single unit of computation in Dagster's task-style API.
• Run launcher: the component that starts runs in processes, containers or pods.
• Staleness: an asset being out of date relative to its upstream assets or code.

Summary

Dagster orchestrates data pipelines by describing the assets they produce and the dependencies between them. That asset-first model gives built-in lineage, partitions, backfills, quality checks and freshness tracking, along with ordinary Python code that is easy to test. It asks you to learn new concepts and has a smaller ecosystem than Apache Airflow, and since July 2026 its company has been part of Prefect. For teams building warehouse and dbt platforms or ML pipelines who want to see what data exists and why, it is a strong choice; see also data lineage for the wider concept.

Related

It belongs to Orchestration in my stack, under Big data. Others in the same group: