Prefect

Overview

Prefect is an open-source orchestration framework for building, scheduling, and monitoring data workflows in Python. It is developed by Prefect Technologies, which also offers a hosted control plane, Prefect Cloud, alongside the self-hostable Prefect Server. Prefect's core pitch is that ordinary Python functions can become orchestrated pipeline components with minimal ceremony: decorate a function with @flow or @task and Prefect adds retries, caching, scheduling, logging, and failure handling around it.

Core concepts

• Flow: the top-level unit of a workflow, defined as a Python function decorated with @flow.
• Task: a discrete unit of work inside a flow, decorated with @task. Tasks can run concurrently and cache results based on inputs.
• Deployment: a flow packaged with the infrastructure and schedule needed to run it (locally, on a schedule, or triggered by an event).
• Work pool / worker: the execution layer that picks up scheduled runs and executes them on infrastructure such as Docker, Kubernetes, or a process pool.

Example flow

Copy code
from prefect import flow, task
import duckdb

@task(retries=2)
def extract_orders():
    return duckdb.sql("SELECT * FROM read_csv_auto('orders.csv')").df()

@task
def load_to_warehouse(df):
    con = duckdb.connect("orders.duckdb")
    con.execute("CREATE OR REPLACE TABLE orders AS SELECT * FROM df")

@flow(name="orders-pipeline")
def orders_pipeline():
    df = extract_orders()
    load_to_warehouse(df)

if __name__ == "__main__":
    orders_pipeline()

Running this file executes the flow directly; deploying it to a schedule adds recurring execution and observability without changing the pipeline code.

Why it matters

Prefect emphasizes a lower-friction authoring experience than more configuration-heavy orchestrators: there's no separate DAG file format to learn, and flows can be tested by simply calling the Python function. Dynamic workflows (where the task graph depends on runtime data, such as looping over an unknown number of files) are also more natural in Prefect than in tools that require the full graph to be defined upfront. In practice, teams often use a Prefect flow as the outer control layer for a data pipeline that runs a DuckDB-based transformation step or triggers a dbt job against a DuckDB or MotherDuck warehouse, relying on Prefect for scheduling and retry logic rather than the compute itself.