Batch processing
Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.
Overview
Batch processing runs a job against a fixed set of accumulated data — for example, all orders placed in the last 24 hours — on a schedule (hourly, nightly, weekly) rather than reacting to each new record individually. It's the traditional model for data warehousing and analytics: extract data from source systems, transform it, and load it into a destination, all as one bounded run.
This contrasts with stream processing, which processes each event as it arrives with continuous, low-latency computation. Batch processing trades immediacy for simplicity, efficiency, and easier reasoning about correctness — a batch job either completes successfully for its whole input or it doesn't, which makes retries, backfills, and debugging more straightforward than in a continuously running stream.
Typical batch workflow
- Extract new or changed records since the last run (often using a timestamp or incrementing ID column)
- Transform the batch (clean, join, aggregate)
- Load the result into a destination table
- Repeat on the next scheduled run
Orchestrators like Airflow, Dagster, or dbt's built-in scheduling coordinate these runs, handle dependencies between jobs, and manage retries on failure.
Incremental batches
Most production batch pipelines don't reprocess the entire dataset on every run — they process only new or changed data since the last successful run, often expressed as an incremental model:
Copy code
-- process only rows newer than the last recorded watermark
SELECT *
FROM raw_events
WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM processed_events);
This keeps each run's processing time proportional to new data volume rather than total data volume, which matters as a dataset grows.
Batch processing with DuckDB
DuckDB is well suited to batch workloads over files: nightly or hourly jobs that read a batch of Parquet or CSV files, transform them with SQL, and write results back out, without needing a persistent server or cluster.
Copy code
COPY (
SELECT
customer_id,
DATE_TRUNC('day', order_date) AS order_day,
SUM(order_amount) AS daily_total
FROM read_parquet('s3://bucket/orders/2026-07-06/*.parquet')
GROUP BY ALL
) TO 's3://bucket/daily_summaries/2026-07-06.parquet' (FORMAT PARQUET);
Because each invocation is a self-contained process reading a batch of files and writing a result, this pattern scales well as a scheduled job (e.g. one per day) without requiring long-running infrastructure.
Why it matters
Most analytics workloads — dashboards refreshed daily, monthly reports, model training pipelines — don't need sub-second latency, which makes batch processing the simpler, cheaper, and more common default. Streaming is reserved for the subset of use cases (fraud detection, real-time monitoring) where low latency is a hard requirement.
Related terms
Lambda architecture is a data processing design that runs a batch layer and a speed layer in parallel to balance completeness and accuracy against low-latency results, merging both into a serving layer.
Kappa architecture →Kappa architecture is a data processing design that uses a single stream-processing pipeline for both real-time and historical data, avoiding the separate batch layer required by Lambda architecture.
Stream processing →Stream processing is the continuous computation of data as individual events arrive, rather than waiting to collect them into a batch. It powers use cases that need results within seconds or milliseconds of an event occurring.
Incremental model →An incremental model is a transformation that processes only new or changed source data on each run and merges it into an existing table, instead of rebuilding the whole table from scratch every time.
data pipeline →A data pipeline is a series of interconnected processes that extract data from various sources, transform it into a usable format, and load it into a…
Vectorized execution →Vectorized execution is a query processing model where each operator processes a batch (a "vector") of values at once, instead of one row at a time, to reduce interpretation overhead and better use modern CPUs.
FAQS
Batch processing runs jobs against accumulated groups of data on a schedule, while stream processing handles each event continuously as it arrives. Batch is simpler and more efficient for workloads that don't need immediate results; streaming trades that simplicity for low latency.
It depends on the use case, but common cadences are hourly, nightly, or daily. The right frequency balances how fresh the data needs to be against the cost and complexity of running more frequent jobs.
Yes. DuckDB's ability to read and write Parquet, CSV, and JSON files directly with SQL, combined with fast in-process analytical execution, makes it a good fit for scheduled batch jobs that transform a batch of files without requiring a persistent cluster.
