Prefect

Overview

Prefect is an open-source workflow orchestrator for Python. You take ordinary Python functions, add a @flow or @task decorator, and Prefect turns them into workflows that can be scheduled, retried, cached, observed and run on remote infrastructure. There is no separate pipeline language and no requirement to declare the whole graph in advance. Loops, conditions and branches are written in normal Python, and Prefect records what actually happened at run time.

The core problem Prefect solves is the gap between a script that works on your laptop and a job that runs reliably every night in production. Production jobs need retries when an API times out, alerts when something fails, a history of runs, logs in one place, a way to run on a schedule or in response to events, and a way to send work to the right machines. Prefect provides these around your existing code. It offers a self-hosted server and a hosted service, Prefect Cloud, both with a web interface.

An analogy: your Python script is a recipe you cook yourself. Prefect is a kitchen manager who keeps a log of every dish, re-tries a step if the oven trips, tells you when something burns and can hand the recipe to whichever cook is free, without rewriting the recipe.

What it is

Where it came from

Prefect was founded by Jeremiah Lowin, who had worked on Apache Airflow and set out to build an orchestrator with a more Pythonic, dynamic model. The first open-source release, then called Prefect Core, appeared in 2019. Prefect 2 was a ground-up redesign that removed the need to define a static graph before running, letting flows be ordinary Python with dynamic control flow.

Prefect 3.0 was announced as an open-source preview in June 2024 and became generally available in September 2024. It added transactions that group tasks into units that can be rolled back, open-sourced the events and automations engine that had previously been limited to Prefect Cloud, and reduced runtime overhead substantially compared with Prefect 2. Prefect is released under the Apache License 2.0. In July 2026 Prefect announced that it was acquiring Dagster Labs, the company behind Dagster, with both products continuing under their own names.

How it works

Writing a flow

You decorate functions. Calling a flow function starts a flow run. Calling a task inside it creates a task run. Prefect wraps each call, records state transitions with the API and applies policies such as retries, timeouts and caching. Tasks can run concurrently with .submit() or .map(), which return futures.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=10)
def fetch(day: str) -> list[dict]:
    ...  # call an API for one day of data

@flow(log_prints=True)
def daily_load(day: str):
    rows = fetch(day)
    print(f"loaded {len(rows)} rows")

if __name__ == "__main__":
    daily_load.serve(name="daily-load", cron="0 2 * * *")

From script to production

1. Run the flow locally like any script; it reports to a local or remote Prefect server so you can see the run in the UI.
2. Create a deployment. The simplest way is flow.serve(), which keeps a process running that executes scheduled runs. For remote infrastructure you use flow.deploy() with a work pool.
3. A schedule, an API call, a UI click or an automation creates a flow run for the deployment.
4. A worker polling the work pool picks up the run and starts it on the configured infrastructure, such as a Docker container or a Kubernetes job.
5. The run reports states, logs and results back to the API. If a task fails, its retry policy applies; if the run fails, automations can send alerts.

Caching and transactions

Tasks can declare a cache policy, for example based on their inputs, so repeated runs with the same inputs reuse a stored result instead of recomputing. Transactions in Prefect 3 group tasks so that, if a later step fails, earlier steps can run rollback hooks, which helps keep side effects consistent.

Events and automations

Prefect emits events for state changes and can accept custom events. Automations watch for patterns, such as "flow run failed" or "no run completed for six hours", and respond by sending notifications, starting deployments or pausing schedules.

Key concepts and terms

A worked example

This scenario is illustrative. A two-person data team pulls sales from three partner APIs each night, cleans the data, loads it into a warehouse and refreshes a forecast. Their cron scripts fail silently about once a week when an API times out.

1. They wrap each API call in a task with three retries and a 30-second delay, and the cleaning and loading steps in further tasks. One flow orchestrates them.
2. The three API tasks are submitted concurrently with .submit(), cutting the extraction step from about 15 minutes to about 6.
3. The cleaning task uses an input-based cache policy, so if loading fails and the flow is re-run, cleaned data from the first attempt is reused.
4. They create a deployment on a Docker work pool with a schedule for 2 a.m. A worker runs on a small virtual machine and starts each run in a fresh container.
5. An automation sends a chat message if a run fails or if no run has completed by 6 a.m.
6. When the forecasting step needs more memory, they move only that flow to a Kubernetes work pool by changing the deployment, without touching the code.

Strengths and limitations

Strengths:

• Very low friction: ordinary Python functions with decorators.
• Dynamic workflows with normal loops, conditions and run-time branching.
• Built-in retries, caching, timeouts, concurrency and transactions.
• Flexible deployment through work pools and workers across many infrastructure types.
• Open-source events and automations, and a hosted option.

Limitations:

• Less focus on data assets and lineage than asset-first tools such as Dagster.
• Breaking changes between major versions have required migration work.
• Self-hosting means running the server, its database and workers yourself.
• Smaller library of ready-made integrations than Airflow.
• Python-centric; other languages need wrapping in Python tasks or containers.

How it compares with alternatives

Practical guidance and best practices

1. Keep tasks small and meaningful, such as one API call or one load, so retries and logs are useful.
2. Give tasks that call external services sensible retries with delays, and set timeouts.
3. Make tasks idempotent so retries and re-runs do not create duplicates.
4. Store credentials in blocks or a secrets manager, never in code.
5. Use serve() for simple setups and work pools with workers when you need isolated or scalable infrastructure.
6. Use cache policies deliberately and give caches an expiry where data changes.
7. Set up automations for failures and for missing runs, not only for errors.
8. Pin the Prefect version and test upgrades in a staging environment.

Common mistakes

• Wrapping an entire script in a single task, so a failure at the end repeats all the work.
• Retrying non-idempotent operations, such as payments or inserts, without safeguards.
• Passing very large objects between tasks without result storage planning.
• Forgetting to run a worker for a work pool, so runs stay scheduled but never start.
• Caching on inputs that do not capture all the data a task depends on.
• Using an orchestrator as the compute engine for heavy processing instead of delegating to a warehouse or cluster.

Frequently asked questions

How is Prefect different from Airflow?

Airflow defines workflows as DAGs of tasks parsed before they run, with a large ecosystem of operators. Prefect treats workflows as ordinary Python executed at run time, so dynamic logic is natural, and it adds retries, caching and state tracking through decorators. Airflow's community and integrations are larger; Prefect is often quicker to adopt for Python teams.

Do I need Prefect Cloud?

No. The open-source Prefect server provides the API and UI, and you can self-host it. Prefect Cloud is a hosted service that adds managed infrastructure and extra team features.

What is the difference between a flow and a task?

A flow is the top-level function that defines a workflow run and can call tasks and other flows. A task is a unit of work inside a flow with its own retry, caching and concurrency settings. Prefect 3 also allows tasks to be called outside flows.

What are work pools and workers?

A work pool describes a type of infrastructure, such as Docker or Kubernetes, and default settings. A worker is a process that polls a work pool and starts flow runs on that infrastructure. Together they separate where code runs from how it is written.

Can Prefect run machine learning pipelines?

Yes. Data preparation, training, evaluation and deployment steps can be tasks or subflows. Heavy training usually runs on dedicated infrastructure, launched through a work pool or by calling an external service.

What changed with the Dagster acquisition?

In July 2026 Prefect announced it was acquiring Dagster Labs. The announcement said both products would continue, with Dagster keeping its brand and open-source project. Prefect users should follow the project's release notes for any roadmap changes.

Glossary

• Control plane: the API, database and UI that track and coordinate runs.
• Idempotent: producing the same effect when repeated.
• Subflow: a flow called from inside another flow.
• Schedule: a cron, interval or calendar rule that creates runs.
• Timeout: a maximum time a task or flow may run.
• Rollback hook: code that undoes a task's side effects when a transaction fails.
• Concurrency limit: a cap on how many runs or tasks of a kind run at once.
• Block: a stored, typed configuration object, such as cloud credentials.

Summary

Prefect turns Python functions into production workflows with decorators, adding retries, caching, transactions, scheduling, events and observability without forcing a separate pipeline language. Deployments, work pools and workers decide where code runs, so the same flow can move from a laptop to containers or Kubernetes. It is less asset-centric than Dagster and has a smaller ecosystem than Apache Airflow, but for Python teams who want dynamic, reliable workflows with little ceremony it is a strong choice. See long-running tasks and Temporal for related patterns.

Related

It belongs to Orchestration in my stack, under Big data. Others in the same group: