Lambda architecture
Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
Overview
Lambda architecture, popularized by Nathan Marz, addresses a tension in large-scale data systems: batch processing produces accurate, complete results but with high latency, while stream processing produces low-latency results that may need later correction. Lambda architecture runs both simultaneously against the same raw data and reconciles them at query time.
The three layers
- Batch layer: stores the immutable, append-only master dataset and periodically recomputes complete, accurate views over all historical data (e.g., a nightly Spark or Hadoop job producing aggregate tables).
- Speed layer: processes only recent data with a stream processor (e.g., Storm, Flink, Kafka Streams) to provide low-latency, approximate views that fill the gap until the batch layer catches up.
- Serving layer: merges batch views and speed views to answer queries, typically favoring the speed layer's results for the most recent time window and the batch layer's results for everything older.
The practical effect: a user querying "total orders today" gets a fast, slightly-provisional number from the speed layer, and once the next batch run completes, that number is replaced by an authoritative one computed from the full dataset.
Tradeoffs
The main criticism of Lambda architecture is that it requires maintaining two separate codebases — one for batch logic, one for streaming logic — that must produce consistent results. This duplication increases operational complexity and is a common source of bugs when the two implementations drift apart. This concern led to the Kappa architecture, which processes everything through a single streaming pipeline and reprocesses history by replaying the stream rather than running a separate batch layer.
Where it's still used
Lambda-style designs remain common where the batch layer's correctness guarantees matter (e.g., financial reporting, compliance data) and where a real-time approximate view is valuable for dashboards or alerting. A batch layer's output is often a set of tables — an analytical engine like DuckDB is a lightweight way to query those materialized batch outputs (e.g., Parquet files) without operating a full cluster.
Related terms
Kappa architecture is a data processing design that uses a single stream-processing pipeline for both real-time and historical data, avoiding the separate batch layer required by Lambda architecture.
Batch processing →Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.
Stream processing →Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
storage layer →The storage layer refers to the component of a data system responsible for persistently storing and managing data.
Semantic layer →A semantic layer is a layer between raw data and end users that defines business metrics, dimensions, and relationships once, so that different tools and teams query consistent, agreed-upon definitions instead of re-deriving them independently.
Medallion architecture →Medallion architecture is a data design pattern that organizes a lakehouse into progressive layers — Bronze (raw), Silver (cleaned), and Gold (business-level aggregates) — improving data quality and structure as it moves through each stage.
FAQS
The name doesn't reference AWS Lambda functions. It comes from the shape of the data flow when diagrammed: raw data forks into a batch layer and a speed layer, then converges again into a serving layer, resembling the Greek letter lambda (λ).
Maintaining two separate processing pipelines (batch and streaming) that must implement the same business logic and produce consistent results. This dual-codebase problem increases development effort and creates a risk of the two paths diverging, which is the main motivation behind the simpler Kappa architecture.
