Kappa architecture
Kappa architecture is a data processing design that uses a single stream-processing pipeline for both real-time and historical data, avoiding the separate batch layer required by Lambda architecture.
Overview
Kappa architecture, proposed by Jay Kreps, simplifies Lambda architecture by eliminating the separate batch layer. Instead of maintaining two codebases — one for batch computation and one for stream computation — Kappa treats everything as a stream. All data, including historical data, is ingested through the same log (typically Kafka or a similar durable, replayable message log), and a single stream-processing job handles both live events and reprocessing.
How reprocessing works
Because the underlying log retains data for a configurable retention period (or indefinitely, in some setups), "batch" recomputation is achieved by replaying the log from an earlier offset through the same streaming job — rather than running a distinct batch pipeline. If processing logic changes or a bug is found, the fix is deployed and the stream is replayed from the point needed to regenerate correct output, then the new results replace the old ones in the serving layer.
Why teams choose it
- One codebase: business logic is written once, in the stream processor, removing the risk of batch and speed layers producing inconsistent results.
- Simpler operations: no need to run and coordinate separate batch infrastructure (e.g., a Hadoop/Spark cluster) alongside the streaming cluster.
- Easier reasoning: since every result — recent or historical — passes through the same code path, debugging and testing are more straightforward.
Tradeoffs
Kappa architecture depends on the log being able to hold and efficiently replay large volumes of historical data, which can be expensive or impractical for very long retention windows. It also assumes the stream processor can handle the throughput of a full historical replay, which may require significant compute for large backfills. Some teams still land stream output into columnar storage (e.g., Parquet, Iceberg) for cost-efficient long-term storage and use a separate query engine like DuckDB for ad hoc analysis over that historical data, even while using a Kappa-style pipeline for the live processing logic itself.
Related terms
Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
Stream processing →Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Batch processing →Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.
Backfill →A backfill is the process of re-running a pipeline over a range of historical data—typically to populate a new table, fix a bug, or apply changed logic to past periods.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
Medallion architecture →Medallion architecture is a data design pattern that organizes a lakehouse into progressive layers — Bronze (raw), Silver (cleaned), and Gold (business-level aggregates) — improving data quality and structure as it moves through each stage.
FAQS
Lambda architecture runs separate batch and speed layers with independent codebases that are merged at query time. Kappa architecture uses a single stream-processing pipeline for both real-time and historical data, reprocessing history by replaying the event log rather than running a separate batch job.
A durable, replayable log (such as Kafka) that retains enough history to support reprocessing, and a stream-processing engine capable of running the same logic against both live events and replayed historical events at the required throughput.
