DOCS
← Back to Glossary
Parquet
Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
Overview
Apache Parquet is a columnar storage file format designed for efficient data processing and analytics. Unlike row-based formats like CSV, Parquet stores data by column rather than by row, which enables better compression and faster querying for analytical workloads. Parquet files also contain metadata about the schema and statistics about the data, allowing query engines to skip reading irrelevant data blocks.
Key Benefits
Parquet provides excellent compression since similar data is stored together in columns. The format supports advanced compression techniques and encoding schemes that work especially well for repeating values. When querying Parquet files, systems like DuckDB can skip reading entire columns that aren't needed for a query (known as column pruning) and can skip reading row groups that don't match filter conditions (known as predicate pushdown).
Usage with DuckDB
DuckDB has native support for reading and writing Parquet files. Here are some common usage patterns:
Reading a Parquet file is as simple as:
SELECT * FROM 'mydata.parquet';
You can write query results to Parquet using:
COPY (SELECT * FROM mytable) TO 'output.parquet' (FORMAT PARQUET);
DuckDB supports configuring Parquet write options like compression codec and row group size:
COPY (SELECT * FROM mytable) TO 'output.parquet' (FORMAT PARQUET, CODEC 'SNAPPY', ROW_GROUP_SIZE 100000);
Integration with Data Lakes
Parquet is a popular format for data lakes built on cloud storage systems like Amazon S3, Azure Data Lake Storage, and Google Cloud Storage. When combined with table formats like Apache Iceberg or Delta Lake, Parquet enables building efficient and scalable data lake architectures.
Related terms
Parquet vs CSV →
Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
ORC →ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
storage →DuckDB uses an efficient columnar storage format optimized for analytical queries.
Reading files in DuckDB →DuckDB can read CSV, Parquet, JSON, and other file formats directly with SQL table functions, whether the files are local, remote over HTTP/S3, or matched in bulk with a glob pattern.
row group →A row group is a fundamental storage concept in columnar databases like DuckDB that represents a horizontal partition of data containing a fixed number of…
COPY statement →The COPY statement bulk-transfers data between a table and an external file, exporting query results to formats like CSV or Parquet, or loading files directly into a table.
