HDFS
HDFS (Hadoop Distributed File System) is a distributed, block-based file system designed to store very large files reliably across clusters of commodity hardware, historically the primary storage layer for Hadoop-based big data systems.
Overview
HDFS splits large files into fixed-size blocks (traditionally 128MB or 256MB) and distributes those blocks across many machines in a cluster, replicating each block (commonly 3x) so that the failure of any single machine doesn't lose data. A NameNode tracks where every block lives, while DataNodes store the actual block data and serve read/write requests.
Design goals and tradeoffs
HDFS was built for high-throughput, sequential access to very large files — the access pattern of batch MapReduce jobs — rather than low-latency random access to many small files. It also tightly couples storage and compute: HDFS DataNodes typically run on the same physical machines as the compute framework (MapReduce, Spark, or Hive) processing the data, so scaling storage and scaling compute happen together.
Why HDFS has receded
Modern data platforms have largely moved away from HDFS toward cloud object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage), which decouples storage from compute, scales more elastically, and is simpler to operate than a self-managed HDFS cluster. Open table formats like Apache Iceberg, Delta Lake, and DuckLake were all designed with object storage as the primary target, and most new lakehouse deployments write directly to S3-compatible storage rather than HDFS.
HDFS and DuckDB
DuckDB does not have built-in HDFS support the way it does for S3-compatible object storage through the httpfs extension. Teams with data still on HDFS typically either migrate it to cloud object storage before querying with DuckDB, or use a Python fsspec-based WebHDFS filesystem registered with DuckDB's Python client to bridge the gap:
Copy code
import fsspec
import duckdb
fs = fsspec.filesystem("webhdfs", host="namenode.example.com", port=50070)
con = duckdb.connect()
con.register_filesystem(fs)
con.sql("SELECT * FROM read_parquet('hdfs://path/to/file.parquet')")
Related terms
Apache Hadoop is an open-source framework for distributed storage and processing of very large datasets across clusters of commodity servers, built around HDFS for storage and MapReduce (or later, YARN-managed engines) for computation.
Object storage →Object storage is a data storage architecture that manages data as discrete objects — each with data, metadata, and a unique identifier — in a flat namespace, accessed over HTTP APIs rather than a traditional file system hierarchy.
storage layer →The storage layer refers to the component of a data system responsible for persistently storing and managing data.
fsspec →fsspec (Filesystem Spec) is a Python library that provides a single, consistent interface for reading and writing files across local disks, cloud object stores, HTTP, and many other storage backends.
Cloud Storage →Cloud storage refers to a model of data storage where digital information is kept on remote servers accessed through the internet, rather than on local hard…
httpfs extension →The httpfs extension is DuckDB's built-in module for reading and writing files directly over HTTP(S) and from S3-compatible object storage, Azure, and Google Cloud Storage, without downloading them first.
FAQS
HDFS is a distributed file system designed to store very large files reliably across a cluster by splitting them into replicated blocks, historically serving as the primary storage layer for Hadoop-based big data processing.
Less so for new deployments. Most organizations have moved to cloud object storage like Amazon S3, Google Cloud Storage, or Azure Blob Storage, which decouples storage from compute and is simpler to operate than a self-managed HDFS cluster.
Not natively — DuckDB's httpfs extension targets HTTP(S) and S3-compatible object storage, not HDFS directly. Common workarounds are migrating data off HDFS to object storage, or registering a Python fsspec-based WebHDFS filesystem with DuckDB's Python client.
