Debezium
Debezium is an open-source distributed platform for change data capture (CDC), which streams row-level insert, update, and delete events out of databases in real time.
Overview
Debezium is an open-source platform for change data capture: it monitors a database's internal transaction/commit log and turns every insert, update, and delete into a structured event, streamed out in real time rather than being discovered later by polling. Debezium was created at Red Hat, which began the project in 2016, and has run as an open-source community project ever since. As of late 2024, Debezium's governance moved from being hosted primarily through Red Hat's infrastructure to the Commonhaus Foundation, a vendor-neutral open-source foundation, while Red Hat remains an active contributor.
How it works
Debezium connectors are most commonly deployed as source connectors on top of Kafka Connect, though a standalone "embedded" mode also exists for running Debezium without Kafka. Each connector reads a specific database's native change log (e.g., MySQL's binlog, Postgres's logical replication slot, MongoDB's oplog) and emits one event per row change, capturing the before and after state of the row.
Copy code
{
"before": {"id": 101, "status": "pending"},
"after": {"id": 101, "status": "shipped"},
"source": {"table": "orders", "db": "shop", "ts_ms": 1715000000000},
"op": "u"
}
Here op: "u" marks an update; Debezium uses c, u, d, and r to represent creates, updates, deletes, and initial snapshot reads respectively.
Why it matters
Log-based CDC avoids two problems with polling a database on a schedule: it misses no intermediate states (a row updated three times between polls still produces three events, not one), and it puts far less load on the source database than repeated SELECT scans. Debezium's event stream is commonly consumed to keep a downstream analytical store continuously in sync with an operational database, feed event-driven microservices, or populate a data lake with an accurate, ordered history of changes. Because Debezium publishes events through Kafka topics (or equivalent), a downstream consumer, such as a job that periodically materializes Parquet files, can be queried directly by an engine like DuckDB, or the events can ultimately land in a MotherDuck database for analysis.
Related terms
Change Data Capture (CDC) is a technique for identifying and streaming row-level inserts, updates, and deletes from a source database as they happen, rather than repeatedly re-reading the whole table.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
UPDATE statement →The UPDATE statement modifies existing rows in a table, setting one or more column values for rows that match an optional WHERE condition.
Stream processing →Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Event-driven architecture →Event-driven architecture is a software design pattern in which services communicate by producing and reacting to events, rather than calling each other directly, enabling loosely coupled, asynchronous systems.
Database replication →Database replication is the process of copying and synchronizing data from one database (the primary) to one or more other databases (replicas), used to improve availability, read scalability, and disaster recovery.
FAQS
The most common and best-supported deployment runs Debezium connectors on Kafka Connect, but Debezium also offers an embedded engine mode for running change capture without a Kafka cluster.
Debezium reads the database's transaction log directly, capturing every change with minimal source load and no missed intermediate states; polling-based ETL queries the current table state periodically and can miss changes that happen and revert between polls.
Debezium originated at Red Hat and moved its governance to the vendor-neutral Commonhaus Foundation in late 2024; Red Hat continues to be an active contributor to the project.
