Schema evolution
Schema evolution is the ability to change a table's structure — adding, removing, renaming, or reordering columns, or widening types — over time without needing to rewrite all existing data or break existing readers.
Overview
Data doesn't stay the same shape forever: new fields get added to source systems, columns get renamed for clarity, and types sometimes need to widen (an integer growing into a bigint, for example). Schema evolution refers to how well a storage system handles these changes gracefully, ideally without a disruptive full rewrite of existing data and without breaking queries or pipelines that expect the old schema.
Schema evolution by format
- Avro — designed from the ground up for evolution: a reader can be given both the schema a record was written with and the schema it wants, and Avro resolves compatible differences (like a missing field with a default value) automatically.
- Parquet — supports limited schema evolution, mainly by merging compatible schemas across files (for example, a newer file has an extra column that older files lack), but it isn't a first-class feature of the format itself.
- Iceberg and Delta Lake — provide full schema evolution as part of the table format: adding, dropping, renaming, and reordering columns are tracked as metadata operations, applied instantly without touching existing data files, because each file's original schema is still recorded and reconciled with the current table schema at read time.
Why table formats matter here
This is one of the clearest examples of what a table format adds over a bare file format: Parquet alone can hold files with slightly different schemas, but it takes something like Iceberg or Delta Lake to formally track schema versions and evolution history at the table level, and to guarantee old queries keep working against new data.
Schema evolution in DuckDB
For native DuckDB tables, ALTER TABLE supports adding, dropping, and renaming columns directly:
Copy code
ALTER TABLE orders ADD COLUMN discount_pct DOUBLE DEFAULT 0.0;
ALTER TABLE orders RENAME COLUMN amount TO order_amount;
When reading multiple Parquet files with slightly different columns, DuckDB's UNION BY NAME lets you combine them safely by column name rather than position:
Copy code
SELECT * FROM read_parquet('data/2025/*.parquet')
UNION BY NAME
SELECT * FROM read_parquet('data/2026/*.parquet');
Related terms
A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Schema registry →A schema registry is a centralized service that stores, versions, and validates data schemas — typically for Avro, Protobuf, or JSON Schema — used to keep producers and consumers in streaming systems like Kafka compatible as schemas change.
ALTER TABLE statement →The ALTER TABLE statement allows you to modify the structure of an existing database table without having to recreate it from scratch.
Avro →Apache Avro is a row-based binary data serialization format that stores a data schema alongside the data, widely used for message serialization in streaming systems like Kafka and for schema evolution in Hadoop-era pipelines.
Apache Iceberg →Apache Iceberg is an open table format that adds ACID transactions, schema evolution, and time travel to large analytic tables stored as files in a data lake.
schema →A schema is a logical container or blueprint that defines how data is organized within a database.
FAQS
Avro was designed for schema evolution from the start, resolving differences between a writer's and reader's schema automatically. Parquet supports limited schema merging across files. Full, table-level schema evolution — tracking add/drop/rename operations without rewriting data — is provided by table formats like Apache Iceberg and Delta Lake.
For native DuckDB tables, ALTER TABLE supports adding, dropping, and renaming columns directly. When reading external files with differing schemas, UNION BY NAME lets you combine them by column name instead of position.
Parquet files are static once written — each just has whatever schema it was written with. Iceberg tracks a table-level schema history in its metadata, reconciling each file's schema with the current one automatically, so schema changes don't require rewriting existing files.
