Table format
A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Overview
A table format sits between raw file formats (Parquet, ORC, Avro) and the query engines that read them. A file format defines how bytes are laid out inside a single file; a table format defines how many files, potentially spread across a whole directory tree, together constitute one logical, evolving table — including its current schema, partitioning scheme, and the exact set of files that make up any given version.
What a table format adds
Without a table format, a "table" on a data lake is just a directory of files, discovered by listing that directory — the approach Hive originally used. That makes concurrent writes unsafe, schema changes destructive, and there's no way to see a consistent snapshot while a write is in progress. A table format solves this with explicit metadata: a chain of files (or, in DuckLake's case, rows in a SQL database) that record every data file, its schema version, and a full history of snapshots, giving:
- ACID transactions — safe concurrent reads and writes
- Schema evolution — adding, renaming, or dropping columns without rewriting data
- Time travel — querying the table as of a past version
- Partition evolution — changing how a table is partitioned going forward without rewriting history
The major table formats
- Apache Iceberg — metadata as chained JSON/Avro files; catalogs like REST, Glue, or Hive Metastore track the current pointer.
- Delta Lake — a JSON transaction log (
_delta_log) alongside Parquet data. - Apache Hudi — similar goals, with strong support for incremental, upsert-heavy pipelines.
- DuckLake — stores metadata in a transactional SQL database (DuckDB, Postgres, or SQLite) instead of files, avoiding the file-listing overhead the other formats incur as metadata grows.
Working with table formats in DuckDB
DuckDB has extensions for the major open table formats, so the same engine can query tables regardless of which format wrote them:
Copy code
LOAD iceberg;
SELECT * FROM iceberg_scan('s3://bucket/warehouse/orders');
LOAD delta;
SELECT * FROM delta_scan('s3://bucket/warehouse/customers');
ATTACH 'ducklake:metadata.ducklake' AS lake (DATA_PATH 's3://bucket/data/');
SELECT * FROM lake.events;
Related terms
Schema evolution is the ability to change a table's structure — adding, removing, renaming, or reordering columns, or widening types — over time without needing to rewrite all existing data or break existing readers.
Apache Iceberg →Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.
Data lakehouse →A data lakehouse is an architecture that combines the low-cost, flexible storage of a data lake with the ACID transactions, schema enforcement, and performance features of a data warehouse.
Delta Lake →Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.
DuckLake →DuckLake is an open table format, created by the DuckDB team, that stores lakehouse metadata in a transactional SQL database instead of files — while keeping the actual data as Parquet files in object storage.
Data lake →A data lake is a centralized repository that stores raw structured, semi-structured, and unstructured data at any scale, in its native format, until it's needed for analysis.
FAQS
A file format (like Parquet or ORC) defines the byte layout of a single file. A table format (like Iceberg, Delta Lake, or DuckLake) defines how a collection of those files together forms one logical, versioned table, adding transactions, schema evolution, and time travel.
The most widely used are Apache Iceberg, Delta Lake, and Apache Hudi, which all store metadata as files alongside the data. DuckLake is a newer format that stores the same kind of metadata in a SQL database instead.
Yes, as long as the engine has support for each format. DuckDB, for example, has separate extensions for Iceberg, Delta Lake, and DuckLake, so it can query tables written by different tools without converting them.
