delta-rs
delta-rs is a native Rust implementation of the Delta Lake protocol, exposed to Python as the `deltalake` package, that lets engines read and write Delta tables without depending on Spark or the JVM.
Overview
delta-rs is a Rust library implementing the Delta Lake transaction protocol from scratch, independent of Spark. It's most commonly used through its Python bindings, distributed as the deltalake package, which lets Python code read, write, and manage Delta tables directly.
Why it exists
Delta Lake originated as a Spark-specific project, which meant any tool wanting to read or write Delta tables needed a JVM and Spark dependency, even for simple operations. delta-rs removes that requirement, giving lightweight, embeddable Delta support to Rust and Python tools — including data frame libraries like Polars and pandas, and orchestration or ETL code that shouldn't need to launch a Spark job just to append rows to a table.
Typical usage
Copy code
from deltalake import DeltaTable, write_deltalake
import pandas as pd
df = pd.DataFrame({"id": [1, 2], "amount": [10.5, 20.0]})
write_deltalake("s3://bucket/warehouse/orders", df, mode="append")
dt = DeltaTable("s3://bucket/warehouse/orders")
result = dt.to_pandas()
Relationship to delta-kernel-rs and DuckDB
delta-rs is a separate project from delta-kernel-rs, a newer Rust library also maintained under the Delta Lake project, designed specifically as a low-level building block for engines to implement their own Delta connectors. DuckDB's delta extension is built on delta-kernel-rs rather than delta-rs directly, but both are part of the same broader move toward engine-agnostic, non-JVM Delta Lake tooling. In practice, this means the same Delta table written with deltalake/delta-rs from a Python script can be read straight back with DuckDB's delta_scan(), with no Spark involved anywhere in the pipeline:
Copy code
INSTALL delta;
LOAD delta;
SELECT * FROM delta_scan('s3://bucket/warehouse/orders');
Related terms
Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Databricks →Databricks is a cloud data and AI platform built by the original creators of Apache Spark, centered on the lakehouse architecture that unifies data lakes and warehousing.
Data lake →A data lake is a centralized repository that stores raw structured, semi-structured, and unstructured data at any scale, in its native format, until it's needed for analysis.
Rust →Rust is a modern, high-performance programming language designed for systems programming and safe concurrency.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
FAQS
delta-rs is a native Rust implementation of the Delta Lake table format, most commonly used through its Python bindings, the deltalake package, to read and write Delta tables without needing Spark or a JVM.
No, they are related but separate Rust projects under the Delta Lake umbrella. delta-rs is a standalone implementation exposed as the deltalake Python package, while delta-kernel-rs is a lower-level library designed for other engines to build their own Delta connectors on top of — DuckDB's delta extension is built on delta-kernel-rs.
Yes. A Delta table written with the deltalake Python package (backed by delta-rs) follows the standard Delta Lake protocol, so DuckDB's delta extension can read it directly with delta_scan(), regardless of which tool wrote it.
