---
title: "HDFS"
description: "HDFS (Hadoop Distributed File System) is a distributed, block-based file system designed to store very large files reliably across clusters of commodity hardware, historically the primary storage layer for Hadoop-based big data systems."
canonical: "https://motherduck.com/glossary/hdfs/"
related:
  - title: "DuckLake Architecture Deep Dive"
    url: "https://motherduck.com/blog/ducklake-architecture-deep-dive/"
  - title: "DuckLake | MotherDuck Docs"
    url: "https://motherduck.com/docs/integrations/file-formats/ducklake/"
  - title: "DuckLake: Making BIG DATA feel small (Coalesce 2025) | MotherDuck"
    url: "https://motherduck.com/videos/ducklake-big-data-small-coalesce-2025/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# HDFS

> HDFS (Hadoop Distributed File System) is a distributed, block-based file system designed to store very large files reliably across clusters of commodity hardware, historically the primary storage layer for Hadoop-based big data systems.

## Overview
HDFS splits large files into fixed-size blocks (traditionally 128MB or 256MB) and distributes those blocks across many machines in a cluster, replicating each block (commonly 3x) so that the failure of any single machine doesn't lose data. A NameNode tracks where every block lives, while DataNodes store the actual block data and serve read/write requests.

## Design goals and tradeoffs
HDFS was built for high-throughput, sequential access to very large files — the access pattern of batch MapReduce jobs — rather than low-latency random access to many small files. It also tightly couples storage and compute: HDFS DataNodes typically run on the same physical machines as the compute framework (MapReduce, Spark, or Hive) processing the data, so scaling storage and scaling compute happen together.

## Why HDFS has receded
Modern data platforms have largely moved away from HDFS toward cloud object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage), which decouples storage from compute, scales more elastically, and is simpler to operate than a self-managed HDFS cluster. Open table formats like Apache Iceberg, Delta Lake, and DuckLake were all designed with object storage as the primary target, and most new lakehouse deployments write directly to S3-compatible storage rather than HDFS.

## HDFS and DuckDB
DuckDB does not have built-in HDFS support the way it does for S3-compatible object storage through the `httpfs` extension. Teams with data still on HDFS typically either migrate it to cloud object storage before querying with DuckDB, or use a Python `fsspec`-based WebHDFS filesystem registered with DuckDB's Python client to bridge the gap:

```python
import fsspec
import duckdb

fs = fsspec.filesystem("webhdfs", host="namenode.example.com", port=50070)
con = duckdb.connect()
con.register_filesystem(fs)
con.sql("SELECT * FROM read_parquet('hdfs://path/to/file.parquet')")
```