Apache Kafka
Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
Overview
Apache Kafka is an open-source distributed event streaming platform. It was originally built at LinkedIn, open-sourced in 2011, and became a top-level Apache Software Foundation project in 2012. Kafka is used to publish, store, and subscribe to streams of events (records) at high throughput, and is a common backbone for real-time data pipelines, event-driven microservices, and log aggregation.
Core concepts
Kafka organizes data into topics, which are split into partitions. Each partition is an append-only, durable, ordered log. Producers write records to topics; consumers read from them, tracking their position (offset) so they can replay or resume. Because records are retained on disk and replicated across brokers, Kafka acts as a durable buffer that decouples the systems that produce data from those that consume it. Consumer groups let multiple consumers share the work of reading a topic in parallel.
Copy code
# Produce and consume messages on a topic with the Kafka CLI
kafka-console-producer.sh --topic events --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic events --from-beginning --bootstrap-server localhost:9092
Where Kafka fits
Kafka is a transport and storage layer for streams, not an analytical query engine. In a typical architecture, Kafka moves events between systems in real time; those events are often written out to object storage or a warehouse in formats like Parquet, where analytical engines then query them. An OLAP engine such as DuckDB (or MotherDuck in the cloud) can query that landed data after the fact, but Kafka itself is about durable, ordered event delivery rather than running SQL analytics.
Related terms
Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Event-driven architecture →Event-driven architecture is a software design pattern in which services communicate by producing and reacting to events, rather than calling each other directly, enabling loosely coupled, asynchronous systems.
Avro →Apache Avro is a row-based binary data serialization format that stores a data schema alongside the data, widely used for message serialization in streaming systems like Kafka and for schema evolution in Hadoop-era pipelines.
Schema registry →A schema registry is a centralized service that stores, versions, and validates data schemas — typically for Avro, Protobuf, or JSON Schema — used to keep producers and consumers in streaming systems like Kafka compatible as schemas change.
Apache Flink →Apache Flink is an open-source distributed engine for stateful stream processing, treating data as continuous streams with low-latency, event-at-a-time computation.
Debezium →Debezium is an open-source distributed platform for change data capture (CDC), which streams row-level insert, update, and delete events out of databases in real time.
FAQS
Kafka is used as a distributed event streaming platform: real-time data pipelines, event-driven microservices, log aggregation, and messaging between systems. It durably stores streams of events in partitioned, append-only logs that producers write to and consumers read from.
A topic is a named stream of records. Each topic is split into partitions, and each partition is an append-only, ordered, durable log replicated across brokers. Consumers track their offset in a partition, which lets them resume or replay records.
