--- title: "Spark, Iceberg & MotherDuck: A Cross-Engine Pipeline | MotherDuck" description: "See a live cross-engine pipeline: Spark transcribes audio and writes to Iceberg, then MotherDuck attaches that catalog directly as a queryable database." canonical: "https://motherduck.com/videos/spark-iceberg-motherduck-pipeline/" --- [BACK TO VIDEOS](/videos/) [Stream](/videos/?category=Stream#videos-and-webinars-library)[Data Pipelines](/videos/?category=Data%20Pipelines#videos-and-webinars-library)[dbt](/videos/?category=dbt#videos-and-webinars-library)[Ecosystem](/videos/?category=Ecosystem#videos-and-webinars-library) # Data takes Flight: Transforming Data with Spark and MotherDuck via Iceberg 2026/09/10 > **TL;DR:** Colin Rogers and Jacob Matson build a cross-engine pipeline live: Spark transcribes and analyzes F1 team radio audio, writes the results to Iceberg, and MotherDuck attaches that Iceberg catalog directly as a queryable database — no data copied, no tables manually declared. ## Storage and compute don't have to live in the same place Apache Iceberg's REST catalog interface means any compute engine that speaks it — Spark, MotherDuck, Databricks, Snowflake — can read and write the same tables. Colin uses AWS Glue's data catalog (with Lake Formation layering an Iceberg REST endpoint on top of S3) as the handoff point: a Spark job on EMR Serverless transcribes F1 radio clips with Whisper, extracts themes with topic modeling, and writes the result to Iceberg. MotherDuck then runs a single `CREATE DATABASE` pointing at that catalog and the tables are just there — attached, not copied, and writable in both directions. ## Why Iceberg here instead of DuckLake Both presenters say outright that DuckLake would probably be a better technical fit for this exact dataset — it's a few thousand rows, nowhere near the partitioned scale Iceberg is designed for. They used Iceberg anyway because it was the easy path from Spark, and because Iceberg's broad compatibility is the actual point of the demo: swap Spark for Databricks or Snowflake and the same MotherDuck attachment still works. ## One monorepo, one pull request The Spark job, the [dbt](/ecosystem/dbt/) project, and the [Dives](/product/dives/) visualization all live in a single repo, orchestrated end to end by a [MotherDuck Flight](https://motherduck.com/product/flights/). Opening a PR spins up a branch-scoped staging version of every piece, including a dive with the branch name suffixed in its title, so a change gets tested across the whole pipeline before it merges rather than staged commit by commit. ## Try the pipeline yourself Colin's full project, including the Spark job, dbt models, and Flight orchestration, is public: [formula-1-data-analysis on GitHub](https://github.com/colin-k-rogers/formula-1-data-analysis). TABLE OF CONTENTS - Storage and compute don't have to live in the same place - Why Iceberg here instead of DuckLake - One monorepo, one pull request - Try the pipeline yourself Start using MotherDuck now! [Try 7 Days Free](https://auth.motherduck.com/authorize?app_source=web&response_type=code&client_id=bza3KWQpxRAFlTlRFXUo29AOg9xD7zcp&redirect_uri=https%3A%2F%2Fapp.motherduck.com%2F&state=STATE&auth_flow=signup&screen_hint=signup&ext-ph_distinct_id=d67498f3-72e4-4135-a7f5-97087772ae51) ## FAQS ### Can I attach an Apache Iceberg catalog to MotherDuck without copying the data? Yes. If your Iceberg tables are registered in a catalog with a REST endpoint — AWS Glue with Lake Formation, Snowflake Polaris, Databricks Unity Catalog, or a self-hosted one — MotherDuck can attach it directly with a single CREATE DATABASE statement. Tables show up automatically without needing individual CREATE TABLE calls, and you can set it up for both reads and writes, not just read-only. ### Why use Iceberg with Spark instead of DuckLake for a pipeline like this? DuckLake probably would have been the better technical fit. The dataset is only a few thousand rows, nowhere near the scale Iceberg is built for. They went with Iceberg because it was the easier path coming from Spark, and because the real point of the demo was cross-engine compatibility: the same MotherDuck attachment works no matter which engine wrote the data. ### What runs the data ingestion in this pipeline, and can it be replaced? Spark on AWS EMR Serverless handles ingestion — it transcribes F1 team radio audio with Whisper, runs topic modeling, and writes to Iceberg. But Spark is an implementation detail. You could swap in Databricks, Snowflake, Ray, Dask, or a custom job on MotherDuck Flights. The REST catalog is the actual integration point, so the compute layer is interchangeable. ### Does dbt work at large data volumes, like petabytes? Yes. dbt doesn't perform transformations itself; your warehouse or query engine does. So dbt's overhead doesn't grow with data volume. What matters at petabyte scale is writing incremental models instead of full refreshes. dbt's automatic DAG building keeps sources and references executing in the right order regardless of how much data flows through. ### Is MotherDuck HIPAA compliant? Yes. MotherDuck is HIPAA compliant and signs Business Associate Agreements as part of its enterprise plans. After the Tower acquisition, MotherDuck Flights (the Python runtime for pipeline orchestration) is covered under that same compliance. ## Related Videos [58:27](/videos/ducklake-time-travel-schema-evolution/)[2026-09-03](/videos/ducklake-time-travel-schema-evolution/) ### [Time Travel & Schema Evolution: DuckLake Handles Change](/videos/ducklake-time-travel-schema-evolution/) Learn how DuckLake time travel, schema evolution, and the change feed work: query any snapshot, undo bad writes, and evolve tables without rewrites. Stream SQL Ecosystem [60:29](/videos/ai-agents-semantic-layer/)[2026-08-26](/videos/ai-agents-semantic-layer/) ### [Do AI Agents Need a Semantic Layer?](/videos/ai-agents-semantic-layer/) Do AI agents need a semantic layer? Evals comparing plain-text context with a Malloy semantic layer show where each wins, and what it costs. Stream AI, ML and LLMs BI & Visualization SQL [63:17](/videos/agentic-data-engineering-stack/)[2026-08-18](/videos/agentic-data-engineering-stack/) ### [Beyond Copilots: We're Building a Data Stack Live with AI Agents](/videos/agentic-data-engineering-stack/) Agentic data engineering in practice: AI agents build dlt ingestion into MotherDuck and a governed Lightdash semantic layer, live in an hour. Stream AI, ML and LLMs Data Pipelines Ecosystem [View all](/videos/)