New: Guides, the context layer for AI agentsRead the blog

START FREE
BACK TO VIDEOS
WebinarSQLMotherDuck Features

The Database Inside Your Lakehouse: A DuckLake Architecture Deep Dive

2026/07/09Featuring:
Alex Monahan
,
Jerel Navarrete

TL;DR: DuckLake stores the catalog and metadata together in a SQL database. Iceberg and Delta scatter metadata across object storage files. This session covers DuckLake's three architectural components — storage, catalog, and compute — including how concurrent writes stay transactionally safe, why inlining fixes the small file problem, and which settings matter for read performance.

Why the metadata lives in a database

Iceberg and Delta already use a database for their catalog, but it holds little more than a pointer to the latest snapshot. The actual metadata sits in layered files on object storage, and a query has to read them sequentially (metadata file, manifest list, manifest file) before it can touch data. At roughly 100 ms per object storage request, that adds up to about half a second of latency before a query even starts. DuckLake puts all of that into the catalog database, so finding the right files is one SQL query that returns in milliseconds. The getting started guide walks through this setup.

The three-component architecture

DuckLake has three pluggable pieces: storage (Parquet files on disk or object storage), a metadata catalog (a DuckDB file locally, Postgres or MotherDuck in production), and compute (DuckDB anywhere, from a laptop to a 192-core serverless instance). The same syntax works from a single-machine lakehouse to multi-user production. You swap components instead of re-architecting. One catalog can serve many compute instances of different sizes at once.

How concurrent writes stay safe

DuckLake uses optimistic concurrency control. Writers push their full Parquet files to object storage first. Those files are invisible to readers because nobody knows a file exists until the catalog says so. Once the data lands, the writer registers the files in a very short catalog transaction with ACID guarantees across tables. Non-conflicting commits retry automatically. That short transaction window is why DuckLake sustains around 100 transactions per second where incumbent formats manage about one.

Where to tune performance

The catalog prunes files using partitioning; compute prunes columns and row groups within files using sorting. Those two settings, plus a scheduled checkpoint for compaction and cleanup, are the main levers for a fast DuckLake. For how DuckLake compares to Iceberg and Delta in the broader stack, see the open lakehouse stack overview.

Questions from the live chat

The chat during the livestream was busy enough that we couldn't get to every question on air. These are the questions attendees typed into the chat, with short answers. Alex followed up on the ones we ran out of time for in a dedicated post, The Database Inside Your Lakehouse: Your DuckLake Questions Answered.

Chat questions answered after the livestream

These came in through the live chat but didn't make it to air. Alex wrote up full answers in the follow-up blog post.

  • Is MotherDuck-managed DuckLake priced per request? No. Pricing is based mostly on compute, with smaller storage costs, like a standard MotherDuck database. See motherduck.com/pricing for current details.
  • Can I run DuckDB compute on a remote server (Postgres catalog + Azure Blob) and let teammates connect over Quack? It's feasible thanks to DuckLake's composability, but a single compute node limits horizontal scaling, and Quack is still in beta with extra setup and authentication to consider.
  • If DuckLake tables perform as well as DuckDB tables, do I still need to "publish" to DuckDB? Using DuckDB as a serving layer is still valid, but DuckLake outperforms other lakehouses on similar workloads, and caching gets you near-DuckDB speed on repeated queries.
  • How does schema evolution work — does it rewrite Parquet files? It doesn't rewrite the whole dataset. Changes apply to new incoming data, and you can set defaults to fill missing historical values.
  • Will DuckLake support primary keys and watermarking? Enforced primary keys are computationally prohibitive in a lakehouse and aren't planned, though unenforced advisory keys could be added.
  • How does DuckLake compare to Snowflake's pg_lake? pg_lake is built on Iceberg, so it inherits Iceberg's low transaction throughput, slower queries, and heavier setup and maintenance.
  • Wouldn't inlining still hit transaction locks? Inlining uses the catalog database's concurrency control, and the deliberately small insert sizes keep lock contention from becoming a problem.
  • Does DuckLake support partitioning for multi-tenancy (per-tenant files and access control)? Partitioning separates data into distinct files and folders. Per-tenant access control is configured in S3 or the catalog database.
  • How do inlined changes flush to Parquet with multiple writers and readers? Small changes and Parquet rewrites proceed fully concurrently without blocking each other.
  • Large-catalog maintenance looked like it needs ~80GB RAM — how do I handle bigger tables? DuckLake spills intermediate work to disk when configured, so it can process larger-than-memory tables.
  • How do table relationships work? There's no explicit relationship definition. You can add column comments (useful for AI agents) or create views with explicit joins.
  • Is the flush mechanism all or nothing? Yes, flush is an atomic all-or-nothing operation.
  • Can DuckLake handle high-concurrency OLTP-style point queries? It isn't optimized for that, but partitioning and sorting act as index equivalents and cut latency versus other lakehouses.
  • Is on-prem enterprise support available? Contact DuckDB Labs directly about enterprise and on-premise support options.
  • What's the licensing model — is DuckDB free commercially? DuckDB and DuckLake are MIT-licensed open source and free for commercial use. MotherDuck is a separate commercial database-as-a-service.
  • How does interoperability with other lakehouse formats work? DuckLake data files are Iceberg-compatible, so Iceberg migration is metadata-only with files staying in place; Delta migration moves the data.
  • Where do I find the maintenance procedure? The DuckLake maintenance Flight is documented on the MotherDuck site.

Chat questions answered live

  • Does it work with all cloud providers, including Azure Blob Storage? Yes. Azure reads and writes are supported (added after the beta).
  • Why no SQL Server catalog yet? No specific timeline, but it's a reasonable candidate, and catalog options will keep expanding.
  • Can DuckLake read DuckDB files in object storage, not just Parquet? Not today. Parquet was chosen for broad compatibility, and the modular design leaves room for it later.
  • How should I think about AWS deployment boundaries (RAM, network)? Multiple compute instances can share one catalog, so you size per workload. Compute needs no local disk, and co-locating compute with storage optimizes latency and cost.
  • When should I pick DuckLake over DuckDB? For multiplayer (concurrent) access and scale (~100GB+). A single DuckDB file is fine up to about 10GB on S3.
  • Can DuckDB/DuckLake Parquet be read without these tools? Yes, it's standard Parquet. Without the catalog, though, you won't know which files are current, so reading through the catalog is the common pattern.
  • How does DuckLake handle metadata bloat from small files? Set a time-travel window for automatic expiry, compact small files, and run a single checkpoint command (schedulable via MotherDuck Flights).
  • Is upsert costly? Plain inserts are fastest, but upserts (selective reads plus deletes and inserts) stay manageable and scalable.
  • VM SSD vs. S3 with caching for small data? DuckDB caches in RAM by default, so SSD usually isn't needed. Disk-caching extensions are in progress.
  • Does catalog type (columnar vs. row) matter for a single client? Catalog work is analytical, so a columnar catalog (MotherDuck DuckDB) helps at scale. Postgres is solid today.
  • Can compute split between cloud and laptop? Yes. MotherDuck dual execution runs light work locally and heavier work in the cloud.
  • Why can MotherDuck's DuckDB do multi-user when plain DuckDB can't? MotherDuck made DuckDB cloud-native and multi-connection. Open-source DuckLake needs Postgres or Quack (beta) for multi-user.
  • Postgres or Quack long-term for concurrency? Use Postgres until Quack leaves beta. Longer term, Quack or MotherDuck catalogs are a strong fit for analytical catalog work.
  • Can I convert existing catalogs without moving Parquet? Iceberg conversion is metadata-only with files staying put. Delta and Fabric likely need data movement.
  • Can I use Looker with DuckDB/DuckLake? Yes, via MotherDuck's Postgres endpoint. See the Looker docs.
  • How does DuckLake avoid locking with concurrent DML? It asks the client to retry if the schema changed mid-commit, and offers extensive tuning for heavy-manipulation environments.
  • Is PostgreSQL or Quack easier to move to MotherDuck? Either works locally; the move is just re-pointing your connections.
  • Any reason not to use DuckLake for time-series sensor data? No, it's an ideal fit. It partitions well, compresses well in Parquet, and the variant type handles semi-structured logs.
  • What's the state of open-source read/write support? Spark, Trino, and DataFusion can read DuckLake. Write support is in development.
  • Will there be UPSERT? It's already supported via the MERGE statement.
  • How are long-running Postgres transactions handled for big inserts? The transaction starts only after the Parquet file is written, so it's very short, which is what enables ~100 transactions per second.
  • Isn't figuring out which files to read also compute work? The initial filter extraction is compute, but the heavy filtering is pushed into SQL WHERE clauses on the catalog.
  • Can I tune the inlining threshold and flush cycle on MotherDuck? Yes, fully via SQL, with scheduling through MotherDuck Flights.
  • Theoretical RPS/RPM limit with a Postgres catalog? No hard number, but ~100 transactions per second versus about one for incumbent lakehouses.
  • Would constant serverless updates spike cost and latency? Instances start in ~100ms and stay warm (seconds for small, about a minute for large), so consecutive requests skip cold starts.
  • Does DuckLake cache S3 Parquet files? Yes, in RAM today. Disk caching is in development.
  • Does the short-transaction answer apply to big inserts on S3? Yes. The target_file_size setting splits large inserts into parallel files.
  • Does the dbt DuckDB adapter work with DuckLake? Yes. Both dbt and SQLMesh have been tested successfully.

FAQS

DuckLake uses optimistic concurrency control. A writer puts its Parquet files in object storage first. Nobody else can see them yet because every read goes through the catalog, and the catalog doesn't know those files exist. Then the writer opens a short catalog transaction, checks whether the global snapshot ID has moved, and commits. If someone else committed in the meantime without changing the schema, DuckLake retries on its own. The slow part — writing files — happens outside the transaction, so the actual catalog commit takes milliseconds. In practice that gets you around 100 transactions per second.

Yes, if you pick a multi-user catalog database. A DuckDB file catalog works for a single user on one machine — fine for local development and CI. For concurrent access you need Postgres, a MotherDuck-managed catalog, or a Quack protocol server (currently in beta). The catalog handles concurrency for all connected compute instances, so multiple people and pipelines can read and write the same DuckLake safely with full multi-table transactional guarantees.

Two signals: concurrency and scale. If multiple people or systems need to read and write the same data at the same time, DuckLake's catalog handles that safely. Plain DuckDB can't. On scale, a single DuckDB file on object storage works fine up to roughly 10GB. Once you're in the hundreds of gigabytes, DuckLake's partitioning and file pruning give real advantages. If you're a single user with small data, DuckDB alone is simpler and plenty fast.

Migration from Iceberg is a metadata-only copy: one function call reads the Avro and JSON metadata files scattered across object storage and compresses them into the DuckLake catalog database. Your Parquet data files stay where they are, since DuckLake's data layer is Iceberg-compatible. Delta is different — there's no metadata-only path, so you move the data itself, usually with insert statements. The getting started guide covers attach options for both managed and bring-your-own-bucket setups.

Tune each layer separately. In the catalog, partitioning by time, customer, or region lets DuckLake skip entire files with a single database query. In the compute layer, sorting your tables lets the engine skip row groups within each file using Parquet footer statistics. Both are table-level settings you can change at any time.

For maintenance, set your time-travel retention and target file sizes, then run the checkpoint command periodically. It flushes inlined data, compacts small files, and vacuums expired ones. Keep compute and storage in the same region on the same cloud — cross-region round trips will kill your latency.

Related Videos

View all