---
title: "Batch processing"
description: "Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives."
canonical: "https://motherduck.com/glossary/batch-processing/"
related:
  - title: "Microbatch: how to supercharge dbt-duckdb with the right incremental model"
    url: "https://motherduck.com/blog/microbatch-dbt-duckdb/"
  - title: "Orchestration | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/orchestration/"
  - title: "Data loading patterns | MotherDuck Docs"
    url: "https://motherduck.com/docs/key-tasks/loading-data-into-motherduck/loading-patterns/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Batch processing

> Batch processing is the execution of data processing jobs on accumulated groups (batches) of records at scheduled intervals, rather than processing each record as it arrives.

## Overview

Batch processing runs a job against a fixed set of accumulated data — for example, all orders placed in the last 24 hours — on a schedule (hourly, nightly, weekly) rather than reacting to each new record individually. It's the traditional model for data warehousing and analytics: extract data from source systems, transform it, and load it into a destination, all as one bounded run.

This contrasts with stream processing, which processes each event as it arrives with continuous, low-latency computation. Batch processing trades immediacy for simplicity, efficiency, and easier reasoning about correctness — a batch job either completes successfully for its whole input or it doesn't, which makes retries, backfills, and debugging more straightforward than in a continuously running stream.

## Typical batch workflow

1. **Extract** new or changed records since the last run (often using a timestamp or incrementing ID column)
2. **Transform** the batch (clean, join, aggregate)
3. **Load** the result into a destination table
4. Repeat on the next scheduled run

Orchestrators like Airflow, Dagster, or dbt's built-in scheduling coordinate these runs, handle dependencies between jobs, and manage retries on failure.

## Incremental batches

Most production batch pipelines don't reprocess the entire dataset on every run — they process only new or changed data since the last successful run, often expressed as an incremental model:

```sql
-- process only rows newer than the last recorded watermark
SELECT *
FROM raw_events
WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM processed_events);
```

This keeps each run's processing time proportional to new data volume rather than total data volume, which matters as a dataset grows.

## Batch processing with DuckDB

DuckDB is well suited to batch workloads over files: nightly or hourly jobs that read a batch of Parquet or CSV files, transform them with SQL, and write results back out, without needing a persistent server or cluster.

```sql
COPY (
  SELECT
    customer_id,
    DATE_TRUNC('day', order_date) AS order_day,
    SUM(order_amount) AS daily_total
  FROM read_parquet('s3://bucket/orders/2026-07-06/*.parquet')
  GROUP BY ALL
) TO 's3://bucket/daily_summaries/2026-07-06.parquet' (FORMAT PARQUET);
```

Because each invocation is a self-contained process reading a batch of files and writing a result, this pattern scales well as a scheduled job (e.g. one per day) without requiring long-running infrastructure.

## Why it matters

Most analytics workloads — dashboards refreshed daily, monthly reports, model training pipelines — don't need sub-second latency, which makes batch processing the simpler, cheaper, and more common default. Streaming is reserved for the subset of use cases (fraud detection, real-time monitoring) where low latency is a hard requirement.