Skip to main content

Querying Files in Amazon S3

Since MotherDuck is hosted in the cloud, one of the benefits of MotherDuck is better and faster interoperability with Amazon S3. MotherDuck's "hybrid mode" automatically routes queries that query Amazon S3 to MotherDuck's execution runtime in the cloud rather than executing these queries locally.

note

MotherDuck supports several cloud storage providers, including Azure and Google Cloud.

MotherDuck supports the DuckDB dialect to query data stored in Amazon S3. Such queries are automatically routed to MotherDuck's cloud execution engines for faster and more efficient execution.

Here are some examples of querying data in Amazon S3:

SELECT * FROM read_parquet('s3://<bucket>/<file>');
SELECT * FROM read_parquet(['s3://<bucket>/<file>', ... ,'s3://<bucket>/<file>']);
SELECT * FROM read_parquet('s3://<bucket>/*');
SELECT * FROM 's3://<bucket>/<directory>/*';
SELECT * FROM iceberg_scan('s3://<bucket>/<directory>', ALLOW_MOVED_PATHS=true);

See Apache Iceberg for more information on reading Iceberg data.

Accessing private files in Amazon S3​

Protected Amazon S3 files require an AWS access key and secret. You can configure MotherDuck in two ways: use CREATE SECRET or use SET commands.

SET s3_region = 'your-s3-region';
SET s3_access_key_id = 'your-aws-access-key-id';
SET s3_secret_access_key = 'your-aws-secret';