Skip to main content
START FREE
BACK TO VIDEOS
StreamData PipelinesdbtEcosystem

Data takes Flight: Transforming Data with Spark and MotherDuck via Iceberg

2026/09/10

TL;DR: Colin Rogers and Jacob Matson build a cross-engine pipeline live: Spark transcribes and analyzes F1 team radio audio, writes the results to Iceberg, and MotherDuck attaches that Iceberg catalog directly as a queryable database — no data copied, no tables manually declared.

Storage and compute don't have to live in the same place

Apache Iceberg's REST catalog interface means any compute engine that speaks it — Spark, MotherDuck, Databricks, Snowflake — can read and write the same tables. Colin uses AWS Glue's data catalog (with Lake Formation layering an Iceberg REST endpoint on top of S3) as the handoff point: a Spark job on EMR Serverless transcribes F1 radio clips with Whisper, extracts themes with topic modeling, and writes the result to Iceberg. MotherDuck then runs a single CREATE DATABASE pointing at that catalog and the tables are just there — attached, not copied, and writable in both directions.

Why Iceberg here instead of DuckLake

Both presenters say outright that DuckLake would probably be a better technical fit for this exact dataset — it's a few thousand rows, nowhere near the partitioned scale Iceberg is designed for. They used Iceberg anyway because it was the easy path from Spark, and because Iceberg's broad compatibility is the actual point of the demo: swap Spark for Databricks or Snowflake and the same MotherDuck attachment still works.

One monorepo, one pull request

The Spark job, the dbt project, and the Dives visualization all live in a single repo, orchestrated end to end by a MotherDuck Flight. Opening a PR spins up a branch-scoped staging version of every piece, including a dive with the branch name suffixed in its title, so a change gets tested across the whole pipeline before it merges rather than staged commit by commit.

Try the pipeline yourself

Colin's full project, including the Spark job, dbt models, and Flight orchestration, is public: formula-1-data-analysis on GitHub.

FAQS

Yes. If your Iceberg tables are registered in a catalog with a REST endpoint — AWS Glue with Lake Formation, Snowflake Polaris, Databricks Unity Catalog, or a self-hosted one — MotherDuck can attach it directly with a single CREATE DATABASE statement. Tables show up automatically without needing individual CREATE TABLE calls, and you can set it up for both reads and writes, not just read-only.

DuckLake probably would have been the better technical fit. The dataset is only a few thousand rows, nowhere near the scale Iceberg is built for. They went with Iceberg because it was the easier path coming from Spark, and because the real point of the demo was cross-engine compatibility: the same MotherDuck attachment works no matter which engine wrote the data.

Spark on AWS EMR Serverless handles ingestion — it transcribes F1 team radio audio with Whisper, runs topic modeling, and writes to Iceberg. But Spark is an implementation detail. You could swap in Databricks, Snowflake, Ray, Dask, or a custom job on MotherDuck Flights. The REST catalog is the actual integration point, so the compute layer is interchangeable.

Yes. dbt doesn't perform transformations itself; your warehouse or query engine does. So dbt's overhead doesn't grow with data volume. What matters at petabyte scale is writing incremental models instead of full refreshes. dbt's automatic DAG building keeps sources and references executing in the right order regardless of how much data flows through.

Yes. MotherDuck is HIPAA compliant and signs Business Associate Agreements as part of its enterprise plans. After the Tower acquisition, MotherDuck Flights (the Python runtime for pipeline orchestration) is covered under that same compliance.

Related Videos

View all