Data orchestration

Overview

Data orchestration is the layer that decides when each step of a data pipeline runs, in what order, and what happens if a step fails. A modern data platform typically has dozens or hundreds of tasks—ingest a source table, transform it, run a data quality check, refresh a dashboard—and most of those tasks depend on one another. An orchestrator encodes those dependencies as a graph, schedules runs (on a cron, an event, or a sensor), retries failures, and gives engineers visibility into what ran, when, and whether it succeeded.

Without orchestration, teams end up chaining cron jobs together and hoping timing works out. That breaks quickly: a transformation might start before its source data has landed, or a failure in one script silently corrupts everything downstream. Orchestration tools make dependencies explicit and observable instead of implicit and fragile.

How it works

Most orchestrators model a pipeline as a directed acyclic graph (DAG): nodes are tasks, edges are dependencies, and the graph has no cycles so execution order is well defined. The orchestrator's scheduler walks the graph, launches tasks whose upstream dependencies have completed, and tracks state (queued, running, succeeded, failed, retrying) for each node.

Common orchestration tools include Airflow, Dagster, Prefect, and dbt's built-in DAG for transformation steps. They typically provide:

• Scheduling — time-based (cron) or event-based (a new file lands, an upstream job finishes)
• Dependency management — a task only runs after its parents succeed
• Retries and alerting — automatic retry with backoff, plus notifications on failure
• Backfills — re-running historical partitions when logic changes
• Observability — run history, logs, and lineage across the whole pipeline

The DuckDB angle

DuckDB is usually the compute engine invoked by orchestrated tasks rather than the orchestrator itself. A common pattern is an Airflow or Dagster task that runs a DuckDB query against files in object storage:

Copy code
import duckdb

con = duckdb.connect("warehouse.duckdb")
con.execute("""
    CREATE OR REPLACE TABLE daily_sales AS
    SELECT *
    FROM read_parquet('s3://bucket/sales/dt=2026-07-06/*.parquet')
""")

Because DuckDB starts instantly and needs no cluster, it's a lightweight building block for orchestrated tasks—each task can spin up a fresh DuckDB process, do its work, and exit, which keeps orchestration graphs simple and each task's resource footprint small. Tools like dbt-duckdb let orchestrators trigger dbt run as a single task that internally manages its own transformation DAG.