Apache Airflow

Overview

Apache Airflow is a workflow orchestration platform for scheduling and monitoring batch data pipelines. Engineers define pipelines as Python code rather than clicking through a UI or writing config files. A pipeline is expressed as a DAG (directed acyclic graph): a collection of tasks with dependencies, but no cycles, so execution always moves forward in a well-defined order. Airflow was created at Airbnb in 2014 to manage the company's growing set of internal data workflows, was open-sourced shortly after, and is now a top-level Apache Software Foundation project with one of the largest communities in the data engineering ecosystem.

Core concepts

• DAG: the overall workflow definition, including schedule interval and default arguments.
• Task / Operator: a single unit of work. Operators wrap common actions (running a Bash command, calling Python, querying a database, moving files) so authors don't reimplement boilerplate.
• Task instance: a specific run of a task for a given DAG run, with its own state (queued, running, success, failed, retried).
• Scheduler: parses DAG files and decides when task instances should run.
• Executor: determines how and where tasks actually execute (locally, via Celery workers, or on Kubernetes).

Example DAG

Copy code
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from datetime import datetime

def run_duckdb_transform():
    import duckdb
    con = duckdb.connect("warehouse.duckdb")
    con.execute("CREATE OR REPLACE TABLE daily_totals AS SELECT date, SUM(amount) FROM raw_orders GROUP BY date")

with DAG(
    dag_id="daily_order_pipeline",
    start_date=datetime(2024, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    extract = BashOperator(task_id="extract_orders", bash_command="python extract_orders.py")
    transform = PythonOperator(task_id="transform_orders", python_callable=run_duckdb_transform)

    extract >> transform

The >> operator declares that transform depends on extract.

Why it matters

Airflow decouples pipeline logic from scheduling infrastructure: retries, backfills, alerting, and dependency management are handled by the platform instead of being hand-rolled in cron scripts. Because tasks are just Python callables or shell commands, a task can run anything, including a lightweight, in-process DuckDB query for local aggregation or a dbt run invocation against a DuckDB or MotherDuck-backed dbt project, without requiring a dedicated compute cluster for small workloads.