Skip to main content

We Classified 100k Rows in 40 Seconds: Introducing prompt_jev()Livestream Sept. 29

← Back to Glossary

Delta Lake

Last updated

Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.

Overview

Delta Lake is an open table format built on top of Parquet files. Every write to a Delta table is recorded as a JSON entry in a transaction log (the _delta_log directory), which lists which Parquet files belong to the current version of the table. Readers use this log to reconstruct a consistent snapshot, which is what gives Delta Lake ACID transactions on top of otherwise immutable files in object storage.

Key features

  • ACID transactions — concurrent readers and writers see a consistent table state via the transaction log.
  • Time travel — you can query a previous version of a table by version number or timestamp.
  • Schema enforcement and evolution — writes are checked against the table's schema, with support for controlled schema changes.
  • Deletion vectors and merge support — efficient row-level updates and deletes without rewriting entire files.

Ecosystem beyond Spark

Delta Lake started as a Spark-specific project, but the protocol is now engine-agnostic. delta-kernel-rs and delta-rs are Rust implementations that let non-Spark engines read and write Delta tables without a JVM, and they power Delta support in tools like DuckDB, Polars, and the Python deltalake package.

DuckDB and Delta Lake

DuckDB's delta extension reads Delta tables directly, including time travel to a specific version:

Copy code
INSTALL delta;
LOAD delta;

SELECT * FROM delta_scan('s3://bucket/warehouse/orders');

-- Attach the table to query a specific version
ATTACH 's3://bucket/warehouse/orders' AS orders (TYPE delta);
SELECT * FROM orders AT (VERSION => 5);

This lets DuckDB and MotherDuck query Delta tables written by Databricks or Spark jobs directly from object storage, without needing a Spark cluster to read them back.

Related terms

delta-rs →

delta-rs is a native Rust implementation of the Delta Lake protocol, exposed to Python as the `deltalake` package, that lets engines read and write Delta tables without depending on Spark or the JVM.

Table format →

A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.

Data lakehouse →

A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.

Databricks →

Databricks is a cloud data and AI platform built by the original creators of Apache Spark, centered on the lakehouse architecture that unifies data lakes and warehousing.

Data lake →

A data lake is a centralized repository that stores raw structured, semi-structured, and unstructured data at any scale, in its native format, until it's needed for analysis.

Apache Iceberg →

Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.

FAQS

Delta Lake keeps a transaction log — a sequence of JSON files in a _delta_log directory — that records which Parquet files make up each version of the table, giving readers a consistent, ACID-compliant view.

No longer. While Delta Lake originated as a Spark project, engine-agnostic Rust implementations like delta-kernel-rs and delta-rs now let engines such as DuckDB, Polars, and the Python deltalake package read and write Delta tables without Spark.

Yes. DuckDB's delta extension provides delta_scan() to read Delta tables directly from local disk or object storage, plus ATTACH ... (TYPE delta) support for querying specific table versions with time travel.

READ NEXT