What is DuckLake? A Simpler Data Lake & Warehouse with DuckDB

A Creator's Critique: The "Aesthetic Problem" with Iceberg

The initial concept of Iceberg was clean. It was founded on a simple, powerful idea: tables could be represented as a collection of files on an object store, with metadata files tracking the state. The beauty was in its self-containment. All you needed was access to the object store, and you could read the table.

But according to Hannes, that philosophical purity was broken when a critical piece of infrastructure was bolted on.

"The aesthetics issue really started when they slapped the catalog server on top to solve some really fundamental issues... they ended up putting this catalog server on top that completely broke with that design philosophy."

— Hannes Mühleisen [11:35]

This wasn't just a minor addition; it was a fundamental architectural shift that introduced a cascade of complexity. The so-called "REST Catalog" requires:

1. A Separate, Always-On Service: A Java service that exposes a REST API.
2. A Backing Database: Typically, a PostgreSQL server to manage the catalog's state.
3. Complex Infrastructure: A fleet of containers and services that must be deployed, monitored, and maintained.

Suddenly, the simple "files on S3" model ballooned into a distributed system. This complexity is most acute for developers tasked with building writers for the format. As Jordan Tigani, CEO of MotherDuck, notes, to build a correct and efficient Iceberg writer, "you basically have to build half of a database."

Hannes goes even further: "You have to write an entire database." A competent writer needs to manage transactions, handle concurrent commits with optimistic locking, perform compaction to prevent the proliferation of small files, and manage snapshot expiration. You're not just writing to a spec; you're implementing the core logic of a database management system.

This has led to a situation Hannes and Jordan compare to the peak of Hadoop. While "knows Iceberg" is becoming a hot skill on a data engineer's resume, the underlying architecture has deep-seated issues. This isn't just a flawed implementation; it's a conceptual problem.

"Iceberg's problems are conceptual, fundamental, and baked in the specification, which is an entirely different way of being wrong."

— Hannes Mühleisen [17:26]

Introducing DuckLake: A First-Principles Data Lake Architecture

This is where "spite engineering," a term Hannes uses to describe a principled refusal to accept flawed solutions, comes into play. The DuckDB team asked a simple, almost heretical question: If we accept that a database is now required to manage the metadata, why not design the entire system around that fact from the beginning?

"Why don't we use a database? ... We are extremely surprised that people think this is a good idea. This is somehow revolutionary because this is, in retrospect... the most obvious thing ever, right?"

— Hannes Mühleisen [24:55]

This question is the foundation of DuckLake. Instead of treating the database as a bolted-on component hidden behind a complex API, DuckLake embraces it as the central nervous system of the lakehouse.

How It Works: A Practical Look at DuckLake

This architectural simplicity translates directly into a superior developer experience. Getting started with a local DuckLake is astonishingly easy.

Copy code
curl https://install.duckdb.org | sh

Copy code
duckdb

Copy code
ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'data_files');
USE my_ducklake;

The MotherDuck Advantage: The Best Hosted DuckLake

At MotherDuck, our mission is to make analytics with DuckDB easy, scalable, and delightful. DuckLake is a core part of that vision.

"Our intent is to be the best hosted Duck Lake. We hope there's other hosted Duck Lakes. We hope that everybody uses Duck Lake."

— Jordan Tigani [48:32]

We are building on DuckLake's powerful foundation to solve the "last mile" problems for developers. A key area is access management. By connecting to MotherDuck, you will be able to manage permissions to your lakehouse using familiar SQL grants. When a query is executed, MotherDuck can generate secure, time-bound signed URLs for the underlying Parquet files, ensuring that even with direct access to your data lake, security is never compromised.

A Call for Simplicity

DuckLake isn't just another standard. It's a fundamental simplification of the lakehouse architecture. It challenges the notion that we need to build complex, bespoke systems to manage what is ultimately structured data.

By leveraging the power, maturity, and transactional integrity of SQL databases for what they do best—managing metadata—DuckLake provides a more robust, more performant, and vastly more developer-friendly path forward. It's a return to first principles, demonstrating that the most elegant solution is often the most obvious one.

Ready to see for yourself? Get started with MotherDuck and DuckLake today. Experience the simplicity and power of a lakehouse built on SQL + Parquet.