START FREE
BACK TO VIDEOS
WebinarData PipelinesEcosystemAI, ML and LLMs

Streaming Made Easy: Practical Postgres CDC with Streamkap + MotherDuck

2026/07/14

TL;DR: Postgres is great until analytical queries start timing out. This session with Jacob Matson (MotherDuck) and Paul Dudley (Streamkap) covers when Postgres change data capture (CDC) is worth it, what tends to break (WAL logs, replication slots, schema changes), and a live Postgres-to-MotherDuck pipeline built with Streamkap.

When you actually need Postgres CDC

Use Postgres for as long as it works. As you scale, analytical queries that scan many rows start to strain a row-based transactional database, and you hit performance and cost problems. That's when it makes sense to offload analytics to a column-oriented engine like MotherDuck, with CDC keeping it in sync in near real time. The what OLAP is and columnar storage guides explain why that workload wants a different database.

What log-based CDC is

Log-based CDC reads a database's change log—every insert, update, and delete—and replays those events into another database. The log already exists, so CDC adds almost no extra load on the source. It also catches deletes that a nightly snapshot would miss.

What breaks, and how to avoid it

The write-ahead log (WAL) is where most of the pain comes from. When you create a replication slot, Postgres holds WAL data until the consumer confirms it's been read. If the connection drops or a slot has low traffic, that WAL keeps growing until it fills your storage and takes down production. Paul's advice: monitor the slot, keep at least three days of WAL headroom, and use heartbeats—artificial traffic that keeps low-traffic slots advancing.

The live demo

Paul streams a Postgres payments table into MotherDuck with Streamkap, kicked off through the Streamkap and MotherDuck MCP servers with Claude. He then builds real-time MotherDuck queries on the streaming data—a payments dashboard and a failure-rate investigation—to show what you can do once the data lands.

FAQS

Start with Postgres. Stay there until it hurts. When your analytical queries — the ones scanning millions of rows — start timing out or dragging down production traffic, move that workload to something column-oriented like MotherDuck. You don't need to migrate anything; Postgres CDC streams every change to the analytical database in near real time. It puts less load on Postgres than a nightly batch job, the data is fresher, and unlike a read replica it gets the analytical work off your primary entirely.

The thing to watch is the write-ahead log. When you create a replication slot, Postgres holds onto WAL data until the consumer acknowledges it. If the connection drops or the slot stalls, that WAL keeps growing and can fill your disk. Before Postgres 16, logical replication only ran on the primary, which made this worse. Monitor your slot lag, keep at least three days of WAL storage headroom, and turn on auto-growth so a weekend outage doesn't page you at 3am.

A heartbeat is fake traffic you generate on purpose to keep a replication slot from going stale. If your slot only tracks a few quiet tables while the rest of the database stays busy, WAL piles up because the slot never signals progress. Streamkap recommends creating a heartbeat table that writes a timestamp on a schedule — every few seconds is fine — so the slot always advances and WAL doesn't accumulate behind it.

Streamkap handles destination schema changes automatically. Add a column in the source and it shows up downstream. Drop a column and the downstream copy stays put but stops receiving data. Change a column's type and Streamkap creates a new column named something like name_datatype alongside the original; if you revert the type, the original column picks back up. The point is that source schema changes don't break the pipeline or anything reading from the destination.

Streamkap runs Kafka and Flink under the hood, built on Debezium, but you never touch any of that directly unless you want to. You connect a source, pick a destination, and it works. Where teams tend to split things: CDC workloads go to Streamkap because it's faster, cheaper, and handles large snapshots better. Fivetran stays in the picture for its API connector library, which is bigger. If you need stream processing beyond replication, Streamkap also lets you use its managed Kafka and Flink directly.

Related Videos

View all