START FREE
BACK TO VIDEOS
StreamAI, ML and LLMsData PipelinesEcosystem

Beyond Copilots: We're Building a Data Stack Live with AI Agents

2026/08/18

TL;DR: Jacob (MotherDuck), Alena (dltHub), and Oliver (Lightdash) build a working data stack live in about an hour — no slides, agents only. An AI-generated dlt pipeline ingests GitHub API data into MotherDuck, and the same schema bootstraps a Lightdash semantic layer and dashboard, so the agent answering questions at the end can't make numbers up.

Agentic data engineering, end to end

AI writing a SQL query was never the hard part. The hard part is everything around it — the ingestion pipeline, the schema wrangling, the models, the metric definitions. This session walks through building all of it with agents, one schema traveling from a REST API to a governed dashboard.

Ingestion without glue code

Alena scaffolds a project with uvx dlthub init and hands Claude Code a two-line prompt: load GitHub commits, issues, and contributors into MotherDuck. The dltHub AI harness does the rest. It finds the right source pattern from a context list of 10,000+ REST APIs, writes a declarative pipeline, debugs it on a small sample, unnests the JSON into relational tables, and deploys it to the dltHub platform on a schedule. Pagination, retries, rate limits, and schema mapping are handled by dlt itself.

Exploring data through the MotherDuck MCP

Jacob connects Claude Code to the MotherDuck MCP server and asks questions in plain language — what's in the database, which issues have been open longest — getting charts back without writing SQL. His point: metrics are archaeology. You discover them by poking around the data, then memorialize them in dashboards.

The semantic layer as a contract

Oliver uses the Lightdash CLI skills to generate the whole BI layer as YAML — models, metrics, charts, and a dashboard — then deploys it without touching the UI. The semantic layer acts as a contract between raw data and consumers. Change a metric definition once in the repo and it fans out to every dashboard and every agent. A lint command keeps the agent honest in a loop, and a governed mode restricts agents to querying only through the semantic layer, with access rules and caching.

Try it yourself

The full demo lives in the dltHub + MotherDuck + Lightdash demo repo, and Lightdash shared a companion resource page for getting started on the BI side.

FAQS

Yes, if you give them enough structure. In this demo, the dltHub AI harness gives Claude Code a specific set of capabilities: source discovery against a list of 10,000+ REST APIs, pipeline scaffolding, debugging on small samples, and deployment. A two-line prompt produces a working dlt pipeline that loads GitHub API data into MotherDuck. The agent validates the data, handles pagination and retries, and deploys the pipeline on a schedule.

A semantic layer sits between your raw warehouse tables and whoever — or whatever — queries them. You define metrics like "total open issues" in versioned YAML files. Consumers query the metric name instead of writing their own SQL against raw tables. When a definition changes, you update it in one place and every dashboard and agent picks it up. This matters more for AI agents than for people. Without a governed semantic layer, an agent will happily invent its own definition of revenue and nobody will notice until the numbers are wrong.

MotherDuck tends to be faster and cheaper for datasets in the tens of terabytes or less. It also has no minimum query latency — BigQuery queries take a few seconds regardless of size, and that adds up fast when an agent fires dozens of small queries in a loop. Postgres is an OLTP database. Analytical dashboards on top of it will struggle without heavy indexing and materialized views.

Yes. The REST API source handles pagination, authentication, backoff and retry (including rate-limit errors), and schema mapping out of the box. It unnests JSON into relational child tables by default. Everything is open-source Python. You define the source declaratively and can customize or disable any of those behaviors.

dlt itself, the Python ingestion library, is open source and free. dltHub is the paid product built around it: managed serverless infrastructure for running pipelines, the AI harness, observability, data quality checks, and transformations under one subscription. The dltHub skills are listed on a public marketplace but require a subscription to run.

Related Videos

View all