DuckLake: Making BIG DATA feel small (Coalesce 2025)

The Big Data Tax

Current cloud data warehouses were designed in 2012 when hardware was much weaker. Their distributed architecture comes with penalties:

• Latency: Small queries take longer than they should due to coordination overhead
• Cost: Network shuffling between nodes isn't free
• Complexity: Scheduling, planning, and routing across nodes adds operational burden

The key insight: "Big compute is dead" (though it's not as catchy as "big data is dead"). Most queries (P99) touch under 256GB of data—well within single-node capability.

DuckDB: Pushing Single-Node Performance

• In-process: Runs inside Python, Node, Go, Rust, and 15+ languages
• Lightweight: 20MB binary, zero dependencies, installs in seconds
• Fast: #1 on ClickBench, beating ClickHouse, Snowflake, Redshift, and BigQuery

DuckLake vs Iceberg Architecture

Key insight: DuckLake uses the same architecture as Snowflake (FoundationDB) and BigQuery (Spanner)—a transactional database for metadata.

5-Minute Lakehouse Demo with dbt

The demo shows setting up a complete lakehouse using:

• A dbt profile configured to use DuckDB with the DuckLake extension
• Postgres as the metadata catalog backend
• Local or cloud storage for the actual data files

Maintenance operations include merging small files, expiring old snapshots, and cleaning up expired files—all callable through dbt run-operation.

Production Considerations

1. Cloud compute: Serverless preferred for simplicity
2. Large instances: Sometimes you need beefy compute for repartitioning or full scans
3. Access control: Lock down your lakehouse
4. Caching: Lakehouse files are immutable—perfect for caching
5. Scheduled maintenance: Automate file compaction and snapshot expiration

MotherDuck: Ducklings of Unusual Size

Real-World Migration

A customer replaced a 5-server distributed cluster (largest AWS instances) running Iceberg with one serverless DuckLake on MotherDuck.

• Migration: Metadata-only (no data copying)
• Iceberg import: Supported for bringing in existing Iceberg data
• Iceberg export: Also supported for interoperability

Key Takeaways

• 10-100x data scale with existing SQL/dbt skills—no new stack or team required
• Instant import from Iceberg—leverage existing data investments
• Local dev parity: Same lakehouse runs on laptop and in production
• Future: Spark connector in development for multi-engine support