---
title: "Apache Spark"
description: "Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines."
canonical: "https://motherduck.com/glossary/apache-spark/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Making PySpark Code Faster with DuckDB"
    url: "https://motherduck.com/blog/making-pyspark-code-faster-with-duckdb/"
  - title: "Data Science & AI | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/data-science-ai/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Apache Spark

> Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.

## Overview

Apache Spark is an open-source engine for large-scale distributed data processing. It originated at UC Berkeley's AMPLab, was donated to the Apache Software Foundation, and became a top-level Apache project in 2014. Spark spreads work across a cluster of machines and keeps intermediate data in memory where possible, which made it substantially faster than earlier disk-based MapReduce for many iterative and interactive workloads.

## Core concepts

Spark's original low-level abstraction is the Resilient Distributed Dataset (RDD), a fault-tolerant collection partitioned across the cluster. Most modern usage is through higher-level DataFrame and Spark SQL APIs, which allow SQL and DataFrame code to be optimized by Spark's Catalyst optimizer. Spark spans several libraries: Spark SQL for structured data, Structured Streaming for stream processing, MLlib for machine learning, and GraphX for graphs. It is written in Scala and offers APIs in Scala, Java, Python (PySpark), and R.

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.read.parquet("s3://data/events/*.parquet")
df.groupBy("event_type").count().orderBy("count", ascending=False).show()
```

## Spark and DuckDB

Spark is designed to scale horizontally across a cluster, which is valuable when data exceeds a single machine. That distribution adds overhead and operational complexity that is unnecessary for workloads that fit on one node. DuckDB is a single-node, in-process OLAP engine that often handles gigabytes-to-hundreds-of-gigabytes workloads faster and more simply than a Spark cluster, and MotherDuck extends it to the cloud. A common pattern is to use Spark for very large distributed pipelines and DuckDB for local development, testing, or medium-scale analytics.

```sql
-- DuckDB: single-node equivalent of the aggregation above
SELECT event_type, COUNT(*) AS n
FROM read_parquet('s3://data/events/*.parquet')
GROUP BY ALL
ORDER BY n DESC;
```