Compression codecs
Compression codecs are algorithms used to reduce the size of data on disk or over the network — common examples include Snappy, Gzip, Zstandard (zstd), and LZ4 — trading off compression ratio against CPU cost and speed.
Overview
A compression codec is an algorithm for encoding data into fewer bytes and decoding it back to its original form. In data engineering, codecs are usually applied per column chunk inside formats like Parquet, ORC, and Avro, or to whole files and network payloads. Choosing a codec is a tradeoff between three things: how much smaller the data gets, how fast it compresses, and how fast it decompresses.
Common codecs
- Snappy — very fast to compress and decompress, with a modest compression ratio. Long the default in Hadoop and early Parquet workloads because CPU time mattered more than storage space.
- Gzip (DEFLATE) — better compression ratio than Snappy, but noticeably slower, especially to compress.
- Zstandard (zstd) — developed at Facebook, offers compression ratios close to or better than Gzip at speeds closer to Snappy, and has become the modern default for many analytical systems.
- LZ4 — prioritizes speed over ratio even more aggressively than Snappy, useful when CPU is the bottleneck and storage is cheap.
Where codecs are applied
File formats like Parquet compress data within each column chunk, independently, which is part of why columnar formats compress so well — similar values sit next to each other. The codec is stored as part of the file's metadata, so any reader knows how to decompress it without extra configuration.
Choosing and setting a codec in DuckDB
DuckDB defaults to zstd compression when writing Parquet files, and lets you override it explicitly:
Copy code
COPY (FROM my_table)
TO 'output.parquet'
(FORMAT parquet, COMPRESSION zstd);
-- Or a lighter-weight codec for faster writes
COPY (FROM my_table)
TO 'output_fast.parquet'
(FORMAT parquet, COMPRESSION snappy);
DuckDB's own internal storage format also applies its own lightweight, per-column compression techniques (like dictionary and run-length encoding) independent of the codec used for exported Parquet files.
Related terms
Data compression encodes data using fewer bits than its original representation, reducing storage footprint and the amount of I/O needed to read it.
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
Run-length encoding →Run-length encoding (RLE) is a compression technique that replaces consecutive repeated values with a single (value, count) pair, shrinking runs of identical data.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
Dictionary encoding →Dictionary encoding is a compression technique that replaces repeated column values with small integer codes, storing the distinct values once in a separate lookup table.
ORC →ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
FAQS
Zstandard (zstd) is a strong general-purpose default, offering a good balance of compression ratio and speed, which is why it's DuckDB's default when writing Parquet. Snappy or LZ4 are better choices when write speed matters more than file size.
Yes, in both directions. Smaller compressed files mean less data to read from disk or network, which usually speeds up queries, but decompression itself takes CPU time — the net effect depends on the codec and whether a workload is I/O-bound or CPU-bound.
Use the COMPRESSION option in a COPY statement, for example: COPY my_table TO 'file.parquet' (FORMAT parquet, COMPRESSION zstd). DuckDB supports zstd, snappy, gzip, and others.
