Data lake

Overview

A data lake is a centralized storage repository that holds large volumes of raw data in its native format — structured tables, semi-structured JSON or logs, and unstructured files like images or text — without requiring a predefined schema. This is often called "schema-on-read": data is loaded as-is, and structure is applied when a query or application reads it, in contrast to a data warehouse's "schema-on-write" model.

Why data lakes exist

Data lakes emerged from the Hadoop ecosystem as a low-cost way to store everything a company generates, rather than only the modeled, cleaned data that fits a warehouse. Today most data lakes live on cloud object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage) instead of HDFS clusters, with data typically written as Parquet, ORC, Avro, JSON, or CSV files organized into directories.

The data swamp problem

Storing raw files cheaply is easy; keeping them usable is hard. Without governance, schema tracking, and transactional guarantees, a data lake can degrade into a "data swamp" — directories of files nobody trusts, with no atomic updates and no easy way to see what changed. This gap is what motivated table formats like Apache Iceberg, Delta Lake, and DuckLake, which add a metadata layer on top of lake files to bring ACID transactions, schema evolution, and time travel — the combination often called a "lakehouse."

Querying a data lake with DuckDB

DuckDB can query files sitting in a data lake directly, without an ETL step into a warehouse. Using the httpfs extension, it reads Parquet or CSV files straight from object storage:

Copy code
INSTALL httpfs;
LOAD httpfs;

SELECT customer_id, SUM(amount) AS total
FROM read_parquet('s3://my-data-lake/orders/*.parquet')
GROUP BY ALL;

This lets analysts explore lake data with SQL immediately, and DuckDB's hive_partitioning option automatically recognizes key=value folder structures common in lakes for partition pruning.