Skip to main content
START FREE
BACK TO VIDEOS
StreamSQLEcosystem

Time Travel & Schema Evolution: DuckLake Handles Change

2026/09/03

TL;DR: Alex Monahan, co-author of the O'Reilly DuckLake book, and Hoyt Emerson of Early Signal work through chapter 4 live: time travel with the AT clause, what snapshots actually cost, schema evolution without rewriting data, and the row-level change feed.

Time travel is an undo button for data

Any DuckLake table accepts an AT clause in the FROM: pass a timestamp or a snapshot ID and you're querying the table as it was. It works per table, so you can anti-join yesterday's table against today's to see what changed, or you can attach the entire lake as of a moment in time. Nothing moves on disk — the catalog just filters to the files that existed then. That's one database query, instead of walking snapshot history file by file on object storage the way older lakehouse formats do.

What snapshots cost

A snapshot is a commit, and DuckLake can keep far more of them than Iceberg because metadata lives in a database instead of separate files. The real cost is retained data: appends make time travel essentially free, updates cost a copy of what changed, and full refreshes are the case to watch — hourly refreshes kept for a week means 168 copies. The fix is tiered retention: keep every transaction for a few days, daily snapshots for a week, weekly beyond that. Expiration is a SQL query on a schedule, and it's deliberately two-step — expire, then delete about a week later, so you keep an undo window.

Schema evolution without rewrites

Add, rename, and drop columns, or widen types (an INTEGER that overflows becomes a BIGINT) without rewriting any existing parquet. Changes must be lossless — no downsizing — which is the compromise that keeps old files readable. Time travel keeps working across schema changes, and you can attach commit messages to snapshots so an audit trail comes free.

The change feed

The data change feed shows every row inserted, updated, or deleted between any two snapshots, including before-and-after values. That's enough to run change data capture off a DuckLake into downstream systems, or to keep a local copy fresh by pulling only what changed. A related trick from the Q&A: copy just the catalog into a local DuckDB file and query the lake read-only from your laptop, leaving the data files where they live.

Keep going

Chapter 4 and the rest of DuckLake: The Definitive Guide are free to download, with new chapters delivered as they're written. New to DuckLake? Start with getting started with the DuckLake table format — it's three commands to try locally.

FAQS

Time travel lets you query or restore any DuckLake table as it existed at a past moment. Add an AT clause to the FROM of any query with a timestamp or snapshot ID, or attach the entire lake as of a point in time to wind everything back at once. It's a metadata operation — no files are moved or rewritten — so it resolves with a single catalog query.

You choose. Deletes are recorded in catalog metadata, so old row versions stay readable until you expire snapshots and compact. Set a retention window that fits your needs, run scheduled expiration beyond it, and use compaction to reclaim storage — including hard-deleting data for GDPR and similar compliance requirements.

It depends on your write pattern. Appends are essentially free — the new rows are data you need anyway. Updates cost a copy of what changed. Full refreshes are the expensive case: each refresh you retain is a full copy of the table, so hourly refreshes kept for a week means 168 copies. A tiered retention policy (every snapshot for days, daily for a week, weekly beyond) keeps costs down without giving up time travel.

You can add, rename, and drop columns, and change column types as long as the change is lossless — widening an INTEGER to a BIGINT works, downsizing doesn't. None of it rewrites existing parquet files; new data flows in with the new schema and old files stay readable. Time travel continues to work across schema changes, and DuckLake tracks the schema history so you can compare tables across versions.

Iceberg keeps three layers of metadata files on object storage, so resolving a snapshot means a series of high-latency reads, and compacting across snapshots is hard — which pressures you to keep few of them. DuckLake puts the catalog and metadata in a regular database, so finding your files is one fast query and keeping thousands of snapshots is fine. The parquet layer is Iceberg-compatible, so migrating an Iceberg table into DuckLake is a single metadata copy.

Related Videos

View all