---
title: "Apache Kafka"
description: "Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale."
canonical: "https://motherduck.com/glossary/apache-kafka/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Streamkap | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/ingestion/streamkap/"
  - title: "Kilo Code + MotherDuck Integration | DuckDB Analytics"
    url: "https://motherduck.com/ecosystem/kilo-code/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Apache Kafka

> Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.

## Overview

Apache Kafka is an open-source distributed event streaming platform. It was originally built at LinkedIn, open-sourced in 2011, and became a top-level Apache Software Foundation project in 2012. Kafka is used to publish, store, and subscribe to streams of events (records) at high throughput, and is a common backbone for real-time data pipelines, event-driven microservices, and log aggregation.

<glossary-callout video="agentic-data-engineering-pipelines-ai" />

## Core concepts

Kafka organizes data into topics, which are split into partitions. Each partition is an append-only, durable, ordered log. Producers write records to topics; consumers read from them, tracking their position (offset) so they can replay or resume. Because records are retained on disk and replicated across brokers, Kafka acts as a durable buffer that decouples the systems that produce data from those that consume it. Consumer groups let multiple consumers share the work of reading a topic in parallel.

```bash
# Produce and consume messages on a topic with the Kafka CLI
kafka-console-producer.sh --topic events --bootstrap-server localhost:9092
kafka-console-consumer.sh --topic events --from-beginning --bootstrap-server localhost:9092
```

## Where Kafka fits

Kafka is a transport and storage layer for streams, not an analytical query engine. In a typical architecture, Kafka moves events between systems in real time; those events are often written out to object storage or a warehouse in formats like Parquet, where analytical engines then query them. An OLAP engine such as DuckDB (or MotherDuck in the cloud) can query that landed data after the fact, but Kafka itself is about durable, ordered event delivery rather than running SQL analytics.