DuckDB secrets

Overview

Before DuckDB introduced a secrets manager, credentials for accessing cloud storage (S3, Azure, GCS) had to be set as individual configuration variables per session, which was repetitive and easy to leak into scripts or notebooks. CREATE SECRET centralizes credential management: secrets are named, typed objects that DuckDB automatically matches to the relevant queries based on type and, optionally, a URL scope.

Creating a secret

Copy code
CREATE SECRET my_s3_secret (
    TYPE s3,
    KEY_ID 'AKIA...',
    SECRET 'wJalrXUtnFEMI...',
    REGION 'us-east-1'
);

By default, secrets created with CREATE SECRET are temporary (in-memory for the session). Adding PERSISTENT writes the secret to disk (in ~/.duckdb/stored_secrets by default) so it's available in future sessions:

Copy code
CREATE PERSISTENT SECRET my_persistent_secret (
    TYPE s3,
    KEY_ID 'AKIA...',
    SECRET 'wJalrXUtnFEMI...'
);

Credential chains and scoping

Rather than hardcoding keys, a secret can use a credential provider chain to pick up credentials from the environment, AWS config files, or an instance's IAM role:

Copy code
CREATE OR REPLACE SECRET s3_secret (
    TYPE s3,
    PROVIDER credential_chain,
    REGION 'eu-west-1'
);

Secrets can also be scoped to specific paths, so that different buckets or accounts use different credentials automatically:

Copy code
CREATE SECRET bucket_a_secret (
    TYPE s3,
    KEY_ID 'key1',
    SECRET 'secret1',
    SCOPE 's3://bucket-a'
);

Why it matters

The secrets manager supports S3, Azure, GCS, and Hugging Face, among others, and is the recommended way to authenticate httpfs-based reads and writes — it keeps credential logic out of individual queries and, with the credential chain provider, avoids embedding raw keys in SQL or scripts at all.