START FREE
BACK TO VIDEOS
WebinarSQLMotherDuck Features

The Database Inside Your Lakehouse: A DuckLake Architecture Deep Dive

2026/07/09Featuring:
Alex Monahan
,
Jerel Navarrete

TL;DR: DuckLake stores the catalog and metadata together in a SQL database. Iceberg and Delta scatter metadata across object storage files. This session covers DuckLake's three architectural components — storage, catalog, and compute — including how concurrent writes stay transactionally safe, why inlining fixes the small file problem, and which settings matter for read performance.

Why the metadata lives in a database

Iceberg and Delta already use a database for their catalog, but it holds little more than a pointer to the latest snapshot. The actual metadata sits in layered files on object storage, and a query has to read them sequentially (metadata file, manifest list, manifest file) before it can touch data. At roughly 100 ms per object storage request, that adds up to about half a second of latency before a query even starts. DuckLake puts all of that into the catalog database, so finding the right files is one SQL query that returns in milliseconds. The getting started guide walks through this setup.

The three-component architecture

DuckLake has three pluggable pieces: storage (Parquet files on disk or object storage), a metadata catalog (a DuckDB file locally, Postgres or MotherDuck in production), and compute (DuckDB anywhere, from a laptop to a 192-core serverless instance). The same syntax works from a single-machine lakehouse to multi-user production. You swap components instead of re-architecting. One catalog can serve many compute instances of different sizes at once.

How concurrent writes stay safe

DuckLake uses optimistic concurrency control. Writers push their full Parquet files to object storage first. Those files are invisible to readers because nobody knows a file exists until the catalog says so. Once the data lands, the writer registers the files in a very short catalog transaction with ACID guarantees across tables. Non-conflicting commits retry automatically. That short transaction window is why DuckLake sustains around 100 transactions per second where incumbent formats manage about one.

Where to tune performance

The catalog prunes files using partitioning; compute prunes columns and row groups within files using sorting. Those two settings, plus a scheduled checkpoint for compaction and cleanup, are the main levers for a fast DuckLake. For how DuckLake compares to Iceberg and Delta in the broader stack, see the open lakehouse stack overview.

FAQS

DuckLake uses optimistic concurrency control. A writer puts its Parquet files in object storage first. Nobody else can see them yet because every read goes through the catalog, and the catalog doesn't know those files exist. Then the writer opens a short catalog transaction, checks whether the global snapshot ID has moved, and commits. If someone else committed in the meantime without changing the schema, DuckLake retries on its own. The slow part — writing files — happens outside the transaction, so the actual catalog commit takes milliseconds. In practice that gets you around 100 transactions per second.

Yes, if you pick a multi-user catalog database. A DuckDB file catalog works for a single user on one machine — fine for local development and CI. For concurrent access you need Postgres, a MotherDuck-managed catalog, or a Quack protocol server (currently in beta). The catalog handles concurrency for all connected compute instances, so multiple people and pipelines can read and write the same DuckLake safely with full multi-table transactional guarantees.

Two signals: concurrency and scale. If multiple people or systems need to read and write the same data at the same time, DuckLake's catalog handles that safely. Plain DuckDB can't. On scale, a single DuckDB file on object storage works fine up to roughly 10GB. Once you're in the hundreds of gigabytes, DuckLake's partitioning and file pruning give real advantages. If you're a single user with small data, DuckDB alone is simpler and plenty fast.

Migration from Iceberg is a metadata-only copy: one function call reads the Avro and JSON metadata files scattered across object storage and compresses them into the DuckLake catalog database. Your Parquet data files stay where they are, since DuckLake's data layer is Iceberg-compatible. Delta is different — there's no metadata-only path, so you move the data itself, usually with insert statements. The getting started guide covers attach options for both managed and bring-your-own-bucket setups.

Tune each layer separately. In the catalog, partitioning by time, customer, or region lets DuckLake skip entire files with a single database query. In the compute layer, sorting your tables lets the engine skip row groups within each file using Parquet footer statistics. Both are table-level settings you can change at any time.

For maintenance, set your time-travel retention and target file sizes, then run the checkpoint command periodically. It flushes inlined data, compacts small files, and vacuums expired ones. Keep compute and storage in the same region on the same cloud — cross-region round trips will kill your latency.

Related Videos

View all