fsspec

Overview

fsspec provides a unified, filesystem-like API in Python that abstracts over dozens of concrete storage backends — local disk, Amazon S3, Google Cloud Storage, Azure Blob Storage, HDFS, HTTP(S), FTP, and more — behind the same interface (open, ls, glob, exists, and so on). Rather than every library implementing its own S3 client, GCS client, etc., tools like pandas, Dask, Zarr, and PyArrow rely on fsspec so that a user can point them at s3://..., gs://..., or https://... paths and have the right backend selected automatically, given the appropriate optional dependency (s3fs, gcsfs, and so on) is installed.

Basic usage

Copy code
import fsspec

fs = fsspec.filesystem("s3")
files = fs.glob("my-bucket/data/*.parquet")

with fsspec.open("s3://my-bucket/data/orders.parquet", "rb") as f:
    data = f.read()

fsspec and DuckDB

DuckDB has its own native, C++-implemented remote filesystem support through the httpfs extension, which covers HTTP(S), S3-compatible object storage, Azure, and Google Cloud Storage, and is generally the fastest option since it avoids the Python layer entirely. For filesystems that httpfs doesn't natively support, DuckDB's Python client can register any fsspec-compliant filesystem via duckdb.register_filesystem(), letting DuckDB query paths on things like Hugging Face Hub, WebHDFS, or other niche or third-party fsspec implementations.

Copy code
import duckdb
from fsspec import filesystem

duckdb.register_filesystem(filesystem("gcs"))
duckdb.sql("SELECT * FROM read_csv('gcs://my-bucket/file.csv')")

Because fsspec filesystems run through Python rather than DuckDB's native C++ I/O path, performance is typically lower than httpfs for the storage systems httpfs already supports directly.