MotherDuck Managed DuckLakes Now in Preview: Scale to Petabytes

How is DuckLake different from Iceberg or Delta Lake?

While Iceberg and Delta Lake pioneered open table formats for massive scale, they suffer from a fundamental performance bottleneck: metadata operations. Every read and write must traverse complex file-based metadata structures, creating latency that compounds at scale.

DuckLake solves this by storing metadata in a transactional database (PostgreSQL, MySQL), delivering:

• 10-100x faster metadata lookups - Database indexes beat file scanning every time
• Instant partition pruning - SQL WHERE clauses on metadata, not file traversal
• Rapid writes at scale - No complex manifest file merging, just database transactions
• Simplified data stack - No additional catalog server, just a standard transactional database that you likely already have organizational expertise in running

Result: Get the same petabyte scale as Iceberg/Delta Lake, but with the snappy performance of a modern data warehouse.

Bonus: DuckLake recognizes that many organizations think of ‘databases’ of inter-related tables, instead of isolated tables, so multi-table ACID transactions are available and you can easily accomplish multi-table schema evolution.

MotherDuck and DuckLake: Warehouse Speed at Lake Scale

Today we're launching a preview of DuckLake—bringing MotherDuck's sub-second query performance to petabyte-scale data lakes.

By using MotherDuck as your DuckLake catalog database, you get:

• Lightning-fast metadata operations powered by MotherDuck's infrastructure
• Seamless scale transitions—start with MotherDuck storage, graduate to DuckLake as you grow
• Unified SQL interface whether querying megabytes or petabytes

Copy code
CREATE SECRET IN MOTHERDUCK (TYPE S3, …);
CREATE DATABASE my_db (TYPE ducklake, DATA_PATH 's3://my-bucket/my-prefix/');

Copy code
CREATE DATABASE my_db (TYPE ducklake);

Access Managed DuckLakes from your Own Cloud (or Laptop)

If you supply your own cloud storage bucket, you can bring your own compute (BYOC) to your DuckLake. Today, this allows you to configure DuckDB to use the DuckLake metadata catalog on MotherDuck, but read and write directly to your cloud storage (let’s say from your AWS Lambda jobs!).

In the DuckDB CLI (as an example), create a secret that provides access to your DATA_PATH:

Copy code
CREATE PRESISTENT SECRET my_secret (
    TYPE S3,
    KEY_ID 'my_s3_access_key',
    SECRET 'my_s3_secret_key',
    REGION 'my-bucket-region'
);

Next, attach the DuckLake to your DuckDB session:

Copy code
ATTACH 'ducklake:md:__ducklake_metadata_<database_name>' AS <alias>;

Now, you can say USE <alias>; to default your DuckDB session to your DuckLake, or just reference the <alias> in your queries. The following will copy a file from a MotherDuck-owned S3 bucket into your DuckLake as a new table.

Copy code
CREATE TABLE <alias>.air_quality AS 
SELECT * FROM 's3://us-prd-motherduck-open-datasets/who_ambient_air_quality/parquet/who_ambient_air_quality_database_version_2024.parquet';

This capability of DuckLakes gets much more interesting when additional data processing frameworks implement support for the DuckLake specification. Support for using DuckLake with Apache Spark is in development.

Time Travel

DuckLake takes consistent snapshots of your data and enables you to query the state of the data as of any snapshot.

Here's an example looking at the state of your customer table 1 week ago:

Copy code
SELECT * FROM customer AT (TIMESTAMP => now() - INTERVAL '1 week');

In order to see the available snapshots, you can use the snapshots() table function:

Copy code
SELECT * FROM snapshots();

You can then run queries against the data at the time of a specific known snapshot:

Copy code
SELECT * FROM customer AT (VERSION => 3);

More information on the time travel semantics is available in the DuckLake time travel and snapshots documentation.

Preview features at a glance

This is an early release of MotherDuck's support for DuckLake. We will continue to expand our capabilities, making your DuckLake faster, easier to use and easier to manage.

Future Support: Our Roadmap

As we work towards GA and beyond, we’ll continue to expand our support for DuckLake at MotherDuck. Since we’re building in the open, we want to share the roadmap with you.

Find us on Slack and reach out to let us know what you think of this preview release and which of the planned features are most important to you. Of course, if there are additional DuckLake capabilities you wish to see, please share those as well.