---
title: "Prefect"
description: "Prefect is a Python-native workflow orchestration framework that lets engineers turn ordinary functions into scheduled, observable, and fault-tolerant data pipelines."
canonical: "https://motherduck.com/glossary/prefect/"
related:
  - title: "Prefect + MotherDuck Integration | DuckDB Analytics"
    url: "https://motherduck.com/ecosystem/prefect/"
  - title: "Prefect | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/orchestration/prefect/"
  - title: "MotherDuck Flights: Agent-Native Data Pipelines | MotherDuck | MotherDuck"
    url: "https://motherduck.com/videos/motherduck-flights-agent-native-data-pipelines/"
gated_asset:
  title: "MotherDuck Flights: Agent-Native Data Pipelines"
  url: "https://motherduck.com/product/flights/"
---

# Prefect

> Prefect is a Python-native workflow orchestration framework that lets engineers turn ordinary functions into scheduled, observable, and fault-tolerant data pipelines.

## Overview

Prefect is an open-source orchestration framework for building, scheduling, and monitoring data workflows in Python. It is developed by Prefect Technologies, which also offers a hosted control plane, Prefect Cloud, alongside the self-hostable Prefect Server. Prefect's core pitch is that ordinary Python functions can become orchestrated pipeline components with minimal ceremony: decorate a function with `@flow` or `@task` and Prefect adds retries, caching, scheduling, logging, and failure handling around it.

<glossary-callout video="agentic-data-engineering-pipelines-ai" />

## Core concepts

- **Flow**: the top-level unit of a workflow, defined as a Python function decorated with `@flow`.
- **Task**: a discrete unit of work inside a flow, decorated with `@task`. Tasks can run concurrently and cache results based on inputs.
- **Deployment**: a flow packaged with the infrastructure and schedule needed to run it (locally, on a schedule, or triggered by an event).
- **Work pool / worker**: the execution layer that picks up scheduled runs and executes them on infrastructure such as Docker, Kubernetes, or a process pool.

## Example flow

```python
from prefect import flow, task
import duckdb

@task(retries=2)
def extract_orders():
    return duckdb.sql("SELECT * FROM read_csv_auto('orders.csv')").df()

@task
def load_to_warehouse(df):
    con = duckdb.connect("orders.duckdb")
    con.execute("CREATE OR REPLACE TABLE orders AS SELECT * FROM df")

@flow(name="orders-pipeline")
def orders_pipeline():
    df = extract_orders()
    load_to_warehouse(df)

if __name__ == "__main__":
    orders_pipeline()
```

Running this file executes the flow directly; deploying it to a schedule adds recurring execution and observability without changing the pipeline code.

## Why it matters

Prefect emphasizes a lower-friction authoring experience than more configuration-heavy orchestrators: there's no separate DAG file format to learn, and flows can be tested by simply calling the Python function. Dynamic workflows (where the task graph depends on runtime data, such as looping over an unknown number of files) are also more natural in Prefect than in tools that require the full graph to be defined upfront. In practice, teams often use a Prefect flow as the outer control layer for a data pipeline that runs a DuckDB-based transformation step or triggers a `dbt` job against a DuckDB or MotherDuck warehouse, relying on Prefect for scheduling and retry logic rather than the compute itself.