Reading files in DuckDB
DuckDB can read CSV, Parquet, JSON, and other file formats directly with SQL table functions, whether the files are local, remote over HTTP/S3, or matched in bulk with a glob pattern.
Overview
One of DuckDB's core strengths is querying files directly as tables without a separate load or import step. Table functions like read_csv, read_parquet, and read_json (along with format-inferring shortcuts) let you point SQL at a file path — local or remote — and query it immediately.
Copy code
SELECT * FROM read_csv('events.csv');
SELECT * FROM read_parquet('orders.parquet');
SELECT * FROM read_json('logs.json');
-- DuckDB also infers the reader from a bare path in some contexts
SELECT * FROM 'orders.parquet';
Reading many files at once
Any of these functions accept a glob pattern or a list of paths to treat multiple files as one logical table:
Copy code
SELECT * FROM read_parquet('data/*.parquet');
SELECT * FROM read_csv(['jan.csv', 'feb.csv', 'mar.csv']);
Passing union_by_name := true to read_parquet or read_csv lets DuckDB reconcile files whose columns don't appear in the same order (or aren't all present), aligning them by column name instead of position.
Reading remote files
With the httpfs extension loaded, the same functions work against HTTP(S) URLs and S3-compatible, Azure, or GCS object storage paths, using credentials configured via CREATE SECRET:
Copy code
SELECT * FROM read_parquet('s3://my-bucket/data/*.parquet');
Format-specific options
Each reader function accepts format-specific parameters — for example, read_csv supports delim, header, columns (to specify types explicitly), and sample_size (controls how much data is scanned for type inference), while read_parquet supports hive_partitioning to recognize key=value directory structures as columns. SUMMARIZE SELECT * FROM read_parquet('orders.parquet') is a quick way to get summary statistics for every column in a file without writing your own aggregation query.
Why it matters
Because these are ordinary SQL table functions, they can be filtered, joined, and aggregated in the same query, and results can be materialized into a table with CREATE TABLE AS SELECT — turning "reading a file" and "querying data" into the same operation.
Related terms
A glob pattern is a wildcard-based string pattern, using characters like `*` and `?`, used to match multiple filenames or paths at once instead of listing them individually.
httpfs extension →The httpfs extension is DuckDB's built-in module for reading and writing files directly over HTTP(S) and from S3-compatible object storage, Azure, and Google Cloud Storage, without downloading them first.
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
COPY statement →The COPY statement bulk-transfers data between a table and an external file, exporting query results to formats like CSV or Parquet, or loading files directly into a table.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
FAQS
No. Functions like read_csv and read_parquet let you query a file's contents directly in a SELECT statement, with no separate load or CREATE TABLE step required, though you can still persist results into a table if you want.
Yes, using the union_by_name option on functions like read_parquet and read_csv, which aligns columns by name rather than assuming every file has identical, identically ordered columns.
