Mage

Overview

Mage, developed as the open-source project mage-ai, is a data pipeline tool that blends the interactivity of a notebook with the structure of a production orchestrator. Instead of writing an entire pipeline as one script, engineers build pipelines from discrete blocks — units of code (Python, SQL, or R) that can be run and tested independently in the browser-based IDE, then chained together into a DAG. Mage targets both classic ETL/ELT use cases and newer AI/ML pipeline needs, such as preparing data for model training or building retrieval-augmented generation workflows.

Core concepts

• Block: a single, independently runnable and testable unit of a pipeline (a data loader, transformer, or exporter).
• Pipeline: an ordered DAG of blocks, visualized and editable in the Mage UI.
• Data loader / transformer / exporter: block types that map to the extract, transform, and load stages of a pipeline.
• Trigger: a schedule, API call, or event that runs a pipeline, similar to triggers in other orchestrators.
• Pipeline runs: each execution is tracked with logs, run history, and block-level status for debugging.

Example block (SQL transformer)

Copy code
-- transformer block: aggregate raw orders
SELECT
  date,
  SUM(amount) AS total_amount
FROM {{ df_1 }}
GROUP BY date

Copy code
# python data loader block
import duckdb

@data_loader
def load_orders(*args, **kwargs):
    con = duckdb.connect("orders.duckdb")
    return con.execute("SELECT * FROM orders").df()

Mage lets SQL and Python blocks reference each other's outputs directly ({{ df_1 }} above refers to the output of a prior block), which keeps intermediate results visible while iterating.

Why it matters

Mage's block-based, notebook-like workflow shortens the iteration loop for building pipelines compared to writing and re-running an entire script on every change, while still producing a schedulable, versioned, deployable pipeline. For local development or smaller datasets, a Mage block can run queries directly against a DuckDB file, giving fast, dependency-light iteration before a pipeline is pointed at a larger warehouse.