Stream processing
Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Overview
Stream processing computes results incrementally as records ('events') arrive, instead of accumulating data and processing it later in a batch. An event might be a page click, a sensor reading, a payment, or a row change captured from a database. A stream processor ingests these events from a message broker or log, applies transformations, aggregations, or joins, and emits updated results continuously.
This is different from batch processing, where a job runs on a bounded dataset (e.g., "yesterday's orders") on a schedule. Streaming systems instead operate on unbounded data: the input never technically ends, and the system has to decide when a window of events is "complete enough" to emit a result, typically using event-time watermarks to handle out-of-order arrivals.
Common building blocks
- Brokers / logs: Apache Kafka, Amazon Kinesis, Google Pub/Sub, Redpanda — durable, ordered logs that decouple producers from consumers.
- Processing engines: Apache Flink, Kafka Streams, Spark Structured Streaming — these run windowed aggregations, stateful joins, and exactly-once or at-least-once delivery semantics over the log.
- Windowing: tumbling, sliding, and session windows group events by time to compute rolling metrics (e.g., "orders per minute").
Where DuckDB fits
DuckDB is an in-process analytical database, not a streaming engine — it does not subscribe to a broker or maintain long-running windowed state. In practice, streaming pipelines land results downstream as files or tables (Parquet, Iceberg, Delta Lake), and DuckDB is used to query those materialized outputs quickly, ad hoc, without standing up a cluster:
Copy code
-- Query the latest micro-batch of streamed events, landed as Parquet by a Kafka connector
SELECT event_type, count(*) AS n
FROM read_parquet('s3://bucket/streaming-sink/dt=2026-07-07/*.parquet')
GROUP BY ALL
ORDER BY n DESC;
This pattern — stream processor writes micro-batches, DuckDB reads and analyzes them — is common when teams want fast, low-overhead analytics on streaming output without running the analytical queries inside the streaming engine itself.
Batch vs. stream
Most organizations use both: streaming for low-latency operational needs (fraud detection, alerting, live dashboards) and batch for cost-efficient, high-throughput historical analysis. The choice depends on how quickly a result needs to be actionable versus how much it costs to keep infrastructure running continuously.
Related terms
Apache Flink is an open-source distributed engine for stateful stream processing, treating data as continuous streams with low-latency, event-at-a-time computation.
Kappa architecture →Kappa architecture is a data processing design that uses a single stream-processing pipeline for both real-time and historical data, avoiding the separate batch layer required by Lambda architecture.
Batch processing →Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
Lambda architecture →Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
Event-driven architecture →Event-driven architecture is a software design pattern in which services communicate by producing and reacting to events, rather than calling each other directly, enabling loosely coupled, asynchronous systems.
FAQS
They're closely related but not identical. Stream processing describes the computation model (processing unbounded event data incrementally). Real-time typically refers to the latency requirement (results within milliseconds to seconds). Most real-time systems use stream processing, but stream processing can also target near-real-time or minute-level latencies.
No. DuckDB is designed for in-process analytical (OLAP) queries over batches or files, not for subscribing to continuous event streams with windowed state. It's commonly used downstream of a streaming engine to query materialized output like Parquet or Iceberg files written by the streaming pipeline.
Batch processing runs on a bounded, finite dataset on a schedule (e.g., hourly or daily jobs). Stream processing runs continuously on unbounded data as events arrive, producing incrementally updated results rather than waiting for a full dataset to be collected.
