Apache Iceberg
Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.
Overview
Apache Iceberg is an open table format originally created at Netflix to fix limitations of Hive tables at scale — most notably that Hive tracks table state by listing directories, which becomes slow and unsafe with concurrent writers. Iceberg instead tracks every data file that makes up a table through a chain of metadata files: a table metadata file points to a manifest list, which points to manifest files, which list the actual data files and their statistics.
Key features
- Snapshot isolation and ACID transactions — every write creates a new snapshot, so readers always see a consistent view and concurrent writers can be safely reconciled.
- Schema evolution — columns can be added, dropped, renamed, or reordered without rewriting data files.
- Hidden partitioning — Iceberg tracks partition transforms (like truncating a timestamp to a day) internally, so queries filter correctly without users needing to know the physical partition layout.
- Time travel — you can query a table as of a past snapshot or timestamp.
Catalogs
Iceberg tables are registered in a catalog that tracks the current metadata pointer for each table. Common catalog implementations include the Iceberg REST catalog, AWS Glue, Hive Metastore, and Nessie.
DuckDB and Apache Iceberg
DuckDB's iceberg extension can read Iceberg tables directly from storage, or by attaching to a REST catalog for full catalog-aware access, including writes:
Copy code
INSTALL iceberg;
LOAD iceberg;
-- Read a table directly from its metadata
SELECT * FROM iceberg_scan('s3://bucket/warehouse/orders');
-- Or attach a REST catalog for catalog-managed tables
ATTACH 'my_warehouse' AS my_iceberg (TYPE iceberg, ENDPOINT 'https://catalog.example.com');
SELECT * FROM my_iceberg.orders;
This lets DuckDB and MotherDuck act as a lightweight, fast query engine over Iceberg tables written by Spark, Flink, or other engines, without standing up a JVM-based cluster.
Related terms
A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
PyIceberg →PyIceberg is the official Python implementation of Apache Iceberg — a pure-Python library for reading, writing, and managing Iceberg tables and catalogs without requiring Spark or a JVM.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Data catalog →A data catalog is a centralized inventory of an organization's datasets, tracking metadata like schema, location, lineage, and ownership so people and systems can discover, understand, and access data.
Apache Hive →Apache Hive is a data warehouse system built on Hadoop that lets users query large datasets stored in HDFS or object storage using HiveQL, a SQL-like language, which Hive translates into distributed jobs.
DuckLake →DuckLake is an open table format, created by the DuckDB team, that stores lakehouse metadata in a transactional SQL database instead of files — while keeping the actual data as Parquet files in object storage.
FAQS
Iceberg replaces directory-listing-based Hive tables with an explicit chain of metadata files that track every data file in a table, enabling safe concurrent writes, consistent snapshots, and schema or partition changes without rewriting data.
Yes. DuckDB's iceberg extension can scan Iceberg tables directly from object storage with iceberg_scan(), or attach to an Iceberg REST catalog with ATTACH ... (TYPE iceberg) to query and write catalog-managed tables.
No. While Spark is a common engine for writing Iceberg tables, many engines can read and write Iceberg, including Trino, Flink, PyIceberg for pure-Python access, and DuckDB via its iceberg extension.
