---
title: "Apache Hadoop"
description: "Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation."
canonical: "https://motherduck.com/glossary/apache-hadoop/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "Data Warehousing How-to | MotherDuck Docs"
    url: "https://motherduck.com/docs/key-tasks/data-warehousing/"
  - title: "Together AI: Scaling a Data Platform while Scaling the Company"
    url: "https://motherduck.com/case-studies/together-ai-scaling-with-motherduck/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# Apache Hadoop

> Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.

## Overview
Apache Hadoop popularized the idea of processing massive datasets by distributing both storage and computation across many ordinary machines, rather than relying on a single, expensive high-end server. Its core components are HDFS (Hadoop Distributed File System) for storage, YARN for cluster resource management, and MapReduce, the original programming model for expressing distributed computations as parallel "map" and "reduce" steps.

## The wider ecosystem
Hadoop grew into a large ecosystem of tools built on top of these primitives: Apache Hive added a SQL-like interface (HiveQL) over HDFS data, Apache HBase provided a distributed NoSQL database, Apache Pig offered a scripting language for data flows, and Apache Spark eventually replaced MapReduce as the dominant processing engine for many Hadoop-era workloads, offering much faster in-memory computation.

## Why its dominance has faded
Running a Hadoop cluster requires significant operational overhead — managing NameNodes, DataNodes, and YARN resource allocation across dozens or hundreds of machines. Cloud object storage decoupled the storage half of Hadoop's value proposition, and modern query engines (Spark, Trino, Snowflake, BigQuery) took over the processing half without requiring a dedicated cluster to be provisioned and maintained. For a large share of workloads that don't actually require distributed computation, single-node engines are now a simpler and often faster alternative.

## DuckDB's relationship to the Hadoop era
DuckDB represents a different point on the spectrum: rather than distributing a query across a cluster, it maximizes what a single machine can do, using a vectorized, columnar execution engine. Because modern hardware can hold and process far more data in memory and on local disk than it could when Hadoop was designed, many analytical workloads that once required a Hadoop cluster now run comfortably — and faster — on a single node with DuckDB, without any of Hadoop's cluster management overhead.