Directed acyclic graph (DAG)
A directed acyclic graph (DAG) is a graph of nodes connected by one-way edges with no cycles, used to represent task dependencies so that execution order is unambiguous.
Overview
A directed acyclic graph, or DAG, is a mathematical structure made of nodes and directed edges (arrows) where following the arrows can never lead back to a node you've already visited. "Directed" means each edge has a direction—A depends on B, not the other way around. "Acyclic" means there are no loops: if you could get from A back to A by following edges, the graph would be invalid for scheduling purposes, because it would imply a task depends on itself, directly or indirectly.
DAGs are the standard way to represent dependencies in data pipelines, build systems, and workflow orchestrators. If task C needs the output of tasks A and B, the DAG has edges A → C and B → C, and any valid execution order runs A and B before C.
Why it matters in data engineering
Orchestration tools like Airflow, Dagster, Prefect, and dbt represent an entire pipeline as a DAG:
- Airflow DAGs are Python objects where each node is an operator (a task) and edges are set with
>>or.set_downstream(). - dbt builds a DAG automatically from
ref()calls between models—ifstg_ordersreferencesraw_orders, dbt knows to buildraw_ordersfirst. - Spark and other query engines compile a query plan into a DAG of stages, scheduling parallel work where the graph allows it and serializing work where it doesn't.
The acyclic property is what makes a pipeline schedulable at all: a topological sort of the DAG produces a valid run order, and any node with no unmet dependencies can run immediately, which is what allows independent branches of a pipeline to execute in parallel.
Copy code
raw_orders ──▶ stg_orders ──▶ fct_orders ──▶ dashboard_orders raw_customers ─▶ stg_customers ─┘
Here stg_orders and stg_customers can build in parallel since neither depends on the other; fct_orders waits for both.
Detecting cycles
If you accidentally introduce a circular dependency—model A references model B, which references model A—most orchestrators will refuse to run and raise a cycle-detection error rather than deadlocking or looping forever. This is a deliberate safety property of DAG-based tools: a cycle has no valid execution order, so it's caught at compile/parse time instead of at runtime.
DuckDB itself doesn't expose DAG concepts directly, but tools built on top of it—like dbt-duckdb—use the same DAG model as any other dbt adapter, resolving ref() dependencies between DuckDB tables/views before executing each model in order.
Related terms
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, where pipelines are defined as directed acyclic graphs (DAGs) in Python.
Data orchestration →Data orchestration is the automated coordination, scheduling, and monitoring of interdependent data tasks—such as extracts, transformations, and loads—so they run in the correct order and recover cleanly from failure.
Graph database →A graph database stores data as nodes and relationships (edges) rather than rows and tables, optimized for traversing and querying densely connected data such as social networks, fraud graphs, or recommendation graphs.
Prefect →Prefect is a Python-native workflow orchestration framework that lets engineers turn ordinary functions into scheduled, observable, and fault-tolerant data pipelines.
Hamilton →Hamilton is an open-source Python micro-framework for defining data and feature transformation pipelines as plain functions, which it automatically assembles into a dependency graph.
Data lineage →Data lineage is the traceable record of where a piece of data came from, what transformations it passed through, and where it's used downstream.
FAQS
A cycle would mean a task depends, directly or indirectly, on its own output—there's no order in which you could run the tasks that satisfies every dependency. Orchestrators require acyclic graphs so a valid run order always exists.
They look similar but serve different purposes. A flowchart typically shows control flow, including branches and loops. A DAG specifically forbids loops and is used to express dependency order, not conditional logic.
Any nodes in the DAG whose upstream dependencies have all completed can run concurrently. The orchestrator computes this from the graph structure rather than requiring you to specify parallelism manually.
