Databricks
Databricks is a cloud data and AI platform built by the original creators of Apache Spark, centered on the lakehouse architecture that unifies data lakes and warehousing.
Overview
Databricks is a cloud-based data and AI platform founded in 2013 by the original creators of Apache Spark, who came out of UC Berkeley's AMPLab. It runs on AWS, Azure, and Google Cloud and is known for popularizing the "lakehouse" architecture, which aims to combine the low-cost, open storage of a data lake with the reliability, performance, and SQL capabilities associated with a data warehouse.
Lakehouse and key components
Several technologies anchor the Databricks platform:
- Apache Spark — the distributed processing engine Databricks was built around and commercializes.
- Delta Lake — an open table format that adds ACID transactions and schema enforcement over files (typically Parquet) in object storage. Delta Lake is developed as the Linux Foundation Delta Lake project.
- Unity Catalog — Databricks' governance layer for data and AI assets, which Databricks open-sourced in 2024.
The lakehouse pattern lets teams keep data in open formats on cheap object storage while still running warehouse-style SQL and machine learning against it.
Databricks and DuckDB
Databricks is a distributed platform (via Spark) aimed at large-scale processing, ML, and multi-user collaboration. DuckDB is an open-source, single-node OLAP engine, with MotherDuck as its serverless cloud counterpart. The two intersect through open formats: DuckDB can read Parquet and Delta Lake data produced by Databricks, which makes it useful for local development, testing, or lightweight analysis of lakehouse data without spinning up a cluster.
Copy code
-- DuckDB reading Parquet from a lakehouse storage path
SELECT event_date, COUNT(*) AS events
FROM read_parquet('s3://lakehouse/events/*.parquet')
GROUP BY ALL
ORDER BY event_date;
Related terms
A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Delta Lake →Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Data lake →A data lake is a centralized repository that stores raw structured, semi-structured, and unstructured data at any scale, in its native format, until it's needed for analysis.
delta-rs →delta-rs is a native Rust implementation of the Delta Lake protocol, exposed to Python as the `deltalake` package, that lets engines read and write Delta tables without depending on Spark or the JVM.
Medallion architecture →Medallion architecture is a data design pattern that organizes a lakehouse into progressive layers — Bronze (raw), Silver (cleaned), and Gold (business-level aggregates) — improving data quality and structure as it moves through each stage.
FAQS
Databricks was founded in 2013 by the original creators of Apache Spark, a team that came out of UC Berkeley's AMPLab. The company commercializes Spark and built its platform around it.
The lakehouse combines the low-cost, open storage of a data lake with the reliability, transactions, and SQL performance associated with a data warehouse. On Databricks it is enabled by open table formats like Delta Lake over files in object storage.
