Write-ahead log (WAL)
A write-ahead log (WAL) is a durability mechanism where every change is recorded in an append-only log before it's applied to the main data files, enabling crash recovery.
Overview
The write-ahead logging protocol is one of the most common techniques for giving a database durability without sacrificing performance. The core rule is simple: a change is written to a sequential log file and flushed to disk before it is applied to the actual data pages, and before the transaction that made it is considered committed. If the process crashes afterward, the database can replay the log on startup to redo any committed changes that hadn't yet made it into the main data files.
How WAL Works
Because appending to a log is a fast, sequential I/O operation, WAL lets a database acknowledge a commit quickly without needing to immediately rewrite scattered data pages on disk. Over time the log grows, so databases periodically run a checkpoint: applying all logged changes to the main data files and then truncating the log, so recovery after a crash only needs to replay the (small) log since the last checkpoint.
DuckDB's WAL
DuckDB writes a <database>.wal file alongside its main .duckdb database file. Every change made outside of a checkpoint is appended to this WAL first. If DuckDB is closed uncleanly, the next time the database is opened it replays the WAL to recover any changes that weren't yet checkpointed. Running CHECKPOINT explicitly merges the WAL into the main database file and clears it:
Copy code
CHECKPOINT;
DuckDB also checkpoints automatically when the WAL grows past a configurable threshold or when the database is closed cleanly.
Related terms
ACID is an acronym describing four properties — Atomicity, Consistency, Isolation, and Durability — that guarantee database transactions are processed reliably.
Change Data Capture (CDC) →Change Data Capture (CDC) is a technique for identifying and streaming row-level inserts, updates, and deletes from a source database as they happen, rather than repeatedly re-reading the whole table.
Kappa architecture →Kappa architecture is a data processing design that uses a single stream-processing pipeline for both real-time and historical data, avoiding the separate batch layer required by Lambda architecture.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Database replication →Database replication is the process of copying and synchronizing data from one database (the primary) to one or more other databases (replicas), used to improve availability, read scalability, and disaster recovery.
Delta Lake →Delta Lake is an open table format, originally developed by Databricks, that adds ACID transactions, schema enforcement, and time travel to Parquet data stored in a data lake.
FAQS
On the next connection, DuckDB replays the .wal file to redo any committed changes that hadn't yet been checkpointed into the main database file, so no committed data is lost.
Run the CHECKPOINT statement, which applies pending WAL changes to the main .duckdb file and truncates the log.
