Apache Hadoop
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
Overview
Apache Hadoop popularized the idea of processing massive datasets by distributing both storage and computation across many ordinary machines, rather than relying on a single, expensive high-end server. Its core components are HDFS (Hadoop Distributed File System) for storage, YARN for cluster resource management, and MapReduce, the original programming model for expressing distributed computations as parallel "map" and "reduce" steps.
The wider ecosystem
Hadoop grew into a large ecosystem of tools built on top of these primitives: Apache Hive added a SQL-like interface (HiveQL) over HDFS data, Apache HBase provided a distributed NoSQL database, Apache Pig offered a scripting language for data flows, and Apache Spark eventually replaced MapReduce as the dominant processing engine for many Hadoop-era workloads, offering much faster in-memory computation.
Why its dominance has faded
Running a Hadoop cluster requires significant operational overhead — managing NameNodes, DataNodes, and YARN resource allocation across dozens or hundreds of machines. Cloud object storage decoupled the storage half of Hadoop's value proposition, and modern query engines (Spark, Trino, Snowflake, BigQuery) took over the processing half without requiring a dedicated cluster to be provisioned and maintained. For a large share of workloads that don't actually require distributed computation, single-node engines are now a simpler and often faster alternative.
DuckDB's relationship to the Hadoop era
DuckDB represents a different point on the spectrum: rather than distributing a query across a cluster, it maximizes what a single machine can do, using a vectorized, columnar execution engine. Because modern hardware can hold and process far more data in memory and on local disk than it could when Hadoop was designed, many analytical workloads that once required a Hadoop cluster now run comfortably — and faster — on a single node with DuckDB, without any of Hadoop's cluster management overhead.
Related terms
HDFS (Hadoop Distributed File System) is a distributed, block-based file system designed to store very large files reliably across clusters of commodity hardware, historically the primary storage layer for Hadoop-based big data systems.
Apache Spark →Apache Spark is an open-source distributed processing engine for large-scale data workloads, using in-memory computation across a cluster of machines.
Apache Hive →Apache Hive is a data warehouse system built on Hadoop that lets users query large datasets stored in HDFS or object storage using HiveQL, a SQL-like language, which Hive translates into distributed jobs.
Apache Flink →Apache Flink is an open-source distributed engine for stateful stream processing, treating data as continuous streams with low-latency, event-at-a-time computation.
Apache Arrow →Apache Arrow is an open-source, cross-language development platform for in-memory columnar data.
Presto →Presto is an open-source distributed SQL query engine, created at Facebook, for running interactive analytical queries across large datasets and multiple data sources.
FAQS
Hadoop's core components are HDFS for distributed storage, YARN for cluster resource management, and MapReduce as the original distributed processing model, later supplemented and largely replaced by Apache Spark for computation.
Hadoop is less commonly chosen for new projects, as cloud object storage and modern single-node or serverless query engines have taken over much of what Hadoop clusters historically provided, with far less operational overhead.
DuckDB runs as a single-node, in-process analytical engine rather than distributing work across a cluster. Since modern hardware can handle far larger datasets than when Hadoop was designed, many workloads that once needed a Hadoop cluster can run faster and more simply on a single machine with DuckDB.
