---
title: "Apache Airflow"
description: "Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, where pipelines are defined as directed acyclic graphs (DAGs) in Python."
canonical: "https://motherduck.com/glossary/apache-airflow/"
related:
  - title: "Airflow | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/orchestration/airflow/"
  - title: "What if a full data pipeline was one prompt away?"
    url: "https://motherduck.com/blog/full-data-pipeline-one-prompt/"
  - title: "AirFlow + MotherDuck Integration | DuckDB Analytics"
    url: "https://motherduck.com/ecosystem/airflow/"
gated_asset:
  title: "MotherDuck Flights: Agent-Native Data Pipelines"
  url: "https://motherduck.com/product/flights/"
---

# Apache Airflow

> Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, where pipelines are defined as directed acyclic graphs (DAGs) in Python.

## Overview

Apache Airflow is a workflow orchestration platform for scheduling and monitoring batch data pipelines. Engineers define pipelines as Python code rather than clicking through a UI or writing config files. A pipeline is expressed as a **DAG** (directed acyclic graph): a collection of tasks with dependencies, but no cycles, so execution always moves forward in a well-defined order. Airflow was created at Airbnb in 2014 to manage the company's growing set of internal data workflows, was open-sourced shortly after, and is now a top-level Apache Software Foundation project with one of the largest communities in the data engineering ecosystem.

<glossary-callout video="agentic-data-engineering-pipelines-ai" />

## Core concepts

- **DAG**: the overall workflow definition, including schedule interval and default arguments.
- **Task / Operator**: a single unit of work. Operators wrap common actions (running a Bash command, calling Python, querying a database, moving files) so authors don't reimplement boilerplate.
- **Task instance**: a specific run of a task for a given DAG run, with its own state (queued, running, success, failed, retried).
- **Scheduler**: parses DAG files and decides when task instances should run.
- **Executor**: determines how and where tasks actually execute (locally, via Celery workers, or on Kubernetes).

## Example DAG

```python
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

def run_duckdb_transform():
    import duckdb
    con = duckdb.connect("warehouse.duckdb")
    con.execute("CREATE OR REPLACE TABLE daily_totals AS SELECT date, SUM(amount) FROM raw_orders GROUP BY date")

with DAG(
    dag_id="daily_order_pipeline",
    start_date=datetime(2024, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    extract = BashOperator(task_id="extract_orders", bash_command="python extract_orders.py")
    transform = PythonOperator(task_id="transform_orders", python_callable=run_duckdb_transform)

    extract >> transform
```

The `>>` operator declares that `transform` depends on `extract`.

## Why it matters

Airflow decouples pipeline logic from scheduling infrastructure: retries, backfills, alerting, and dependency management are handled by the platform instead of being hand-rolled in cron scripts. Because tasks are just Python callables or shell commands, a task can run anything, including a lightweight, in-process DuckDB query for local aggregation or a `dbt run` invocation against a DuckDB or MotherDuck-backed dbt project, without requiring a dedicated compute cluster for small workloads.