Delta Lake
Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.
Overview
Delta Lake is an open table format built on top of Parquet files. Every write to a Delta table is recorded as a JSON entry in a transaction log (the _delta_log directory), which lists which Parquet files belong to the current version of the table. Readers use this log to reconstruct a consistent snapshot, which is what gives Delta Lake ACID transactions on top of otherwise immutable files in object storage.
Key features
- ACID transactions — concurrent readers and writers see a consistent table state via the transaction log.
- Time travel — you can query a previous version of a table by version number or timestamp.
- Schema enforcement and evolution — writes are checked against the table's schema, with support for controlled schema changes.
- Deletion vectors and merge support — efficient row-level updates and deletes without rewriting entire files.
Ecosystem beyond Spark
Delta Lake started as a Spark-specific project, but the protocol is now engine-agnostic. delta-kernel-rs and delta-rs are Rust implementations that let non-Spark engines read and write Delta tables without a JVM, and they power Delta support in tools like DuckDB, Polars, and the Python deltalake package.
DuckDB and Delta Lake
DuckDB's delta extension reads Delta tables directly, including time travel to a specific version:
Copy code
INSTALL delta;
LOAD delta;
SELECT * FROM delta_scan('s3://bucket/warehouse/orders');
-- Attach the table to query a specific version
ATTACH 's3://bucket/warehouse/orders' AS orders (TYPE delta);
SELECT * FROM orders AT (VERSION => 5);
This lets DuckDB and MotherDuck query Delta tables written by Databricks or Spark jobs directly from object storage, without needing a Spark cluster to read them back.
Related terms
delta-rs is a native Rust implementation of the Delta Lake protocol, exposed to Python as the `deltalake` package, that lets engines read and write Delta tables without depending on Spark or the JVM.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Databricks →Databricks is a cloud data and AI platform built by the original creators of Apache Spark, centered on the lakehouse architecture that unifies data lakes and warehousing.
Data lake →A data lake is a centralized repository that stores raw structured, semi-structured, and unstructured data at any scale, in its native format, until it's needed for analysis.
Apache Iceberg →Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.
FAQS
Delta Lake keeps a transaction log — a sequence of JSON files in a _delta_log directory — that records which Parquet files make up each version of the table, giving readers a consistent, ACID-compliant view.
No longer. While Delta Lake originated as a Spark project, engine-agnostic Rust implementations like delta-kernel-rs and delta-rs now let engines such as DuckDB, Polars, and the Python deltalake package read and write Delta tables without Spark.
Yes. DuckDB's delta extension provides delta_scan() to read Delta tables directly from local disk or object storage, plus ATTACH ... (TYPE delta) support for querying specific table versions with time travel.
