httpfs extension

Overview

httpfs is a core DuckDB extension that adds support for querying files over the network as if they were local. It covers plain HTTP(S) URLs as well as S3-compatible object storage (AWS S3 and compatible services like MinIO, Cloudflare R2, and Google Cloud Storage's S3-compatible API), and, alongside a related extension, Azure Blob Storage. It uses HTTP range requests so DuckDB can read only the byte ranges it actually needs (for example, Parquet footers and specific row groups) instead of downloading an entire file.

Installing and using it

Recent versions of DuckDB autoload httpfs the first time it's needed, but it can also be installed and loaded explicitly:

Copy code
INSTALL httpfs;
LOAD httpfs;

SELECT *
FROM read_parquet('s3://my-bucket/data/orders.parquet');

SELECT *
FROM read_csv('https://example.com/data/events.csv');

Credentials

Reading from private S3, Azure, or GCS locations requires credentials. DuckDB's secrets manager is the standard way to configure them:

Copy code
CREATE SECRET my_s3_secret (
    TYPE s3,
    KEY_ID 'my_key',
    SECRET 'my_secret',
    REGION 'us-east-1'
);

Once created, matching queries against s3:// paths automatically use the secret without needing to pass credentials in every query.

Why it matters

httpfs is central to DuckDB's role as a lightweight analytical engine for the modern data lake: it lets DuckDB query Parquet, CSV, or JSON files sitting in cloud object storage directly, including across many files matched by a glob pattern, without standing up a separate service or copying data locally first.