Apache Spark
Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Overview
Apache Spark is an open-source engine for large-scale distributed data processing. It originated at UC Berkeley's AMPLab, was donated to the Apache Software Foundation, and became a top-level Apache project in 2014. Spark spreads work across a cluster of machines and keeps intermediate data in memory where possible, which made it substantially faster than earlier disk-based MapReduce for many iterative and interactive workloads.
Core concepts
Spark's original low-level abstraction is the Resilient Distributed Dataset (RDD), a fault-tolerant collection partitioned across the cluster. Most modern usage is through higher-level DataFrame and Spark SQL APIs, which allow SQL and DataFrame code to be optimized by Spark's Catalyst optimizer. Spark spans several libraries: Spark SQL for structured data, Structured Streaming for stream processing, MLlib for machine learning, and GraphX for graphs. It is written in Scala and offers APIs in Scala, Java, Python (PySpark), and R.
Copy code
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("example").getOrCreate()
df = spark.read.parquet("s3://data/events/*.parquet")
df.groupBy("event_type").count().orderBy("count", ascending=False).show()
Spark and DuckDB
Spark is designed to scale horizontally across a cluster, which is valuable when data exceeds a single machine. That distribution adds overhead and operational complexity that is unnecessary for workloads that fit on one node. DuckDB is a single-node, in-process OLAP engine that often handles gigabytes-to-hundreds-of-gigabytes workloads faster and more simply than a Spark cluster, and MotherDuck extends it to the cloud. A common pattern is to use Spark for very large distributed pipelines and DuckDB for local development, testing, or medium-scale analytics.
Copy code
-- DuckDB: single-node equivalent of the aggregation above
SELECT event_type, COUNT(*) AS n
FROM read_parquet('s3://data/events/*.parquet')
GROUP BY ALL
ORDER BY n DESC;
Related terms
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
Databricks →Databricks is a cloud data and AI platform built by the original creators of Apache Spark, centered on the lakehouse architecture that unifies data lakes and warehousing.
Apache Arrow →Apache Arrow is an open-source, cross-language development platform for in-memory columnar data.
Apache Flink →Apache Flink is an open-source distributed engine for stateful stream processing, treating data as continuous streams with low-latency, event-at-a-time computation.
PyIceberg →PyIceberg is the official Python implementation of Apache Iceberg — a pure-Python library for reading, writing, and managing Iceberg tables and catalogs without requiring Spark or a JVM.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
FAQS
Spark is used for large-scale distributed data processing: ETL pipelines, batch and streaming analytics, and machine learning across a cluster of machines. It provides Spark SQL, Structured Streaming, and MLlib, with APIs in Scala, Java, Python, and R.
Use DuckDB when your workload fits on a single machine. It avoids the overhead and operational complexity of a distributed cluster and is often faster and simpler for gigabyte-to-hundreds-of-gigabyte analytics, local development, and testing. Spark is warranted when data truly exceeds one node.
