Apache Flink
Apache Flink is an open-source distributed engine for stateful stream processing, treating data as continuous streams with low-latency, event-at-a-time computation.
Overview
Apache Flink is an open-source framework and distributed engine for stateful computations over data streams. It originated from the Stratosphere research project (led out of TU Berlin), was renamed Flink and donated to the Apache Software Foundation, and became a top-level Apache project in 2014. Flink is built for stream processing first: it processes events as they arrive, one at a time, rather than as scheduled micro-batches, which enables low-latency, continuous computation.
Streaming model
Flink treats bounded (batch) data as a special case of unbounded (streaming) data, so the same engine handles both. Key capabilities include:
- Event-time processing — computing results based on when events actually occurred, with watermarks to handle out-of-order and late data.
- Stateful operators — maintaining large, fault-tolerant state, with consistency provided by periodic checkpoints.
- Windowing — grouping events into time or count windows for aggregation.
Flink exposes layered APIs, including the DataStream API and Flink SQL / Table API for expressing stream and batch logic in SQL.
Copy code
-- Flink SQL: a tumbling window aggregation over a stream
SELECT window_start, COUNT(*) AS events
FROM TABLE(
TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '1' MINUTE))
GROUP BY window_start;
Where Flink fits
Flink is a processing engine for continuous, real-time transformation and aggregation of streams, often reading from and writing to systems like Apache Kafka. It is distinct from analytical query engines used for querying data at rest. Streaming pipelines built with Flink frequently land their output in object storage or a warehouse, where an OLAP engine such as DuckDB or MotherDuck can then run analytical SQL over the results.
Related terms
Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Apache Hadoop →Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
Lambda architecture →Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
Batch processing →Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.
FAQS
Flink was designed as a true streaming engine that processes events one at a time as they arrive, whereas Spark historically processed streams as micro-batches. Flink treats batch as a special case of streaming, and emphasizes event-time processing, large fault-tolerant state, and low-latency computation.
Stateful stream processing means operators maintain state across events, such as running aggregates or window contents. Flink stores this state in a fault-tolerant way and uses periodic checkpoints to recover consistently after failures, enabling accurate continuous results over unbounded streams.
