Kestra

Overview

Kestra is an open-source orchestration platform for building, scheduling, and monitoring data and application workflows. It is developed by Kestra Technologies. Unlike orchestrators such as Airflow or Prefect where pipelines are written as Python code, Kestra workflows are defined declaratively as YAML files, with a plugin system that supplies the actual task implementations (querying a database, calling an HTTP API, running a script, moving files between storage systems). Kestra also has a UI that can generate and edit this YAML directly, which lowers the barrier for non-Python users to build workflows.

Core concepts

• Flow: a YAML document describing a workflow: its identifier, namespace, triggers, and ordered list of tasks.
• Task: a single step in a flow, backed by a plugin (e.g., io.kestra.plugin.jdbc, io.kestra.plugin.scripts.python).
• Trigger: what starts a flow — a schedule, a webhook, a new file landing in storage, or a message on a queue. Kestra's trigger model is built around being event-driven, not just time-based.
• Namespace: a way of organizing and scoping flows, similar to a project or folder.

Example flow

Copy code
id: daily_orders_pipeline
namespace: analytics

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 6 * * *"

tasks:
  - id: extract
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - python extract_orders.py

  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import duckdb
      con = duckdb.connect("warehouse.duckdb")
      con.execute("CREATE OR REPLACE TABLE daily_totals AS SELECT date, SUM(amount) FROM raw_orders GROUP BY date")

Why it matters

Because each Kestra task is really a call out to a plugin, a task can shell out to run a Python script that queries a local DuckDB file, or trigger a dbt run against a DuckDB or MotherDuck-backed project, while Kestra itself just handles scheduling, retries, concurrency limits, and observability around that step. The declarative, language-agnostic design also makes it easier for teams with mixed skill sets (not everyone comfortable writing Python DAGs) to contribute workflows through the UI.