---
title: "Stream processing"
description: "Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring."
canonical: "https://motherduck.com/glossary/stream-processing/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Streamkap | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/ingestion/streamkap/"
  - title: "Practical Postgres CDC with Streamkap + MotherDuck | MotherDuck"
    url: "https://motherduck.com/videos/postgres-cdc-streamkap-motherduck/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Stream processing

> Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.

## Overview

Stream processing computes results incrementally as records ('events') arrive, instead of accumulating data and processing it later in a batch. An event might be a page click, a sensor reading, a payment, or a row change captured from a database. A stream processor ingests these events from a message broker or log, applies transformations, aggregations, or joins, and emits updated results continuously.

This is different from batch processing, where a job runs on a bounded dataset (e.g., "yesterday's orders") on a schedule. Streaming systems instead operate on unbounded data: the input never technically ends, and the system has to decide when a window of events is "complete enough" to emit a result, typically using event-time watermarks to handle out-of-order arrivals.

## Common building blocks

- **Brokers / logs**: Apache Kafka, Amazon Kinesis, Google Pub/Sub, Redpanda — durable, ordered logs that decouple producers from consumers.
- **Processing engines**: Apache Flink, Kafka Streams, Spark Structured Streaming — these run windowed aggregations, stateful joins, and exactly-once or at-least-once delivery semantics over the log.
- **Windowing**: tumbling, sliding, and session windows group events by time to compute rolling metrics (e.g., "orders per minute").

## Where DuckDB fits

DuckDB is an in-process analytical database, not a streaming engine — it does not subscribe to a broker or maintain long-running windowed state. In practice, streaming pipelines land results downstream as files or tables (Parquet, Iceberg, Delta Lake), and DuckDB is used to query those materialized outputs quickly, ad hoc, without standing up a cluster:

```sql
-- Query the latest micro-batch of streamed events, landed as Parquet by a Kafka connector
SELECT event_type, count(*) AS n
FROM read_parquet('s3://bucket/streaming-sink/dt=2026-07-07/*.parquet')
GROUP BY ALL
ORDER BY n DESC;
```

This pattern — stream processor writes micro-batches, DuckDB reads and analyzes them — is common when teams want fast, low-overhead analytics on streaming output without running the analytical queries inside the streaming engine itself.

## Batch vs. stream

Most organizations use both: streaming for low-latency operational needs (fraud detection, alerting, live dashboards) and batch for cost-efficient, high-throughput historical analysis. The choice depends on how quickly a result needs to be actionable versus how much it costs to keep infrastructure running continuously.