---
title: "fsspec"
description: "fsspec (Filesystem Spec) is a Python library that provides a single, consistent interface for reading and writing files across local disks, cloud object stores, HTTP, and many other storage backends."
canonical: "https://motherduck.com/glossary/fsspec/"
related:
  - title: "DuckLake | MotherDuck Docs"
    url: "https://motherduck.com/docs/concepts/ducklake/"
  - title: "DuckDB, the great federator?"
    url: "https://motherduck.com/blog/duckdb-the-great-federator/"
  - title: "Querying Files in Amazon S3 | MotherDuck Docs"
    url: "https://motherduck.com/docs/key-tasks/cloud-storage/querying-s3-files/"
gated_asset:
  title: "DuckLake on MotherDuck"
  url: "https://motherduck.com/product/ducklake/"
---

# fsspec

> fsspec (Filesystem Spec) is a Python library that provides a single, consistent interface for reading and writing files across local disks, cloud object stores, HTTP, and many other storage backends.

## Overview

fsspec provides a unified, filesystem-like API in Python that abstracts over dozens of concrete storage backends — local disk, Amazon S3, Google Cloud Storage, Azure Blob Storage, HDFS, HTTP(S), FTP, and more — behind the same interface (`open`, `ls`, `glob`, `exists`, and so on). Rather than every library implementing its own S3 client, GCS client, etc., tools like pandas, Dask, Zarr, and PyArrow rely on fsspec so that a user can point them at `s3://...`, `gs://...`, or `https://...` paths and have the right backend selected automatically, given the appropriate optional dependency (`s3fs`, `gcsfs`, and so on) is installed.

## Basic usage

```python
import fsspec

fs = fsspec.filesystem("s3")
files = fs.glob("my-bucket/data/*.parquet")

with fsspec.open("s3://my-bucket/data/orders.parquet", "rb") as f:
    data = f.read()
```

## fsspec and DuckDB

DuckDB has its own native, C++-implemented remote filesystem support through the `httpfs` extension, which covers HTTP(S), S3-compatible object storage, Azure, and Google Cloud Storage, and is generally the fastest option since it avoids the Python layer entirely. For filesystems that `httpfs` doesn't natively support, DuckDB's Python client can register any fsspec-compliant filesystem via `duckdb.register_filesystem()`, letting DuckDB query paths on things like Hugging Face Hub, WebHDFS, or other niche or third-party fsspec implementations.

```python
import duckdb
from fsspec import filesystem

duckdb.register_filesystem(filesystem("gcs"))
duckdb.sql("SELECT * FROM read_csv('gcs://my-bucket/file.csv')")
```

Because fsspec filesystems run through Python rather than DuckDB's native C++ I/O path, performance is typically lower than `httpfs` for the storage systems `httpfs` already supports directly.