Data compression
Data compression encodes data using fewer bits than its original representation, reducing storage footprint and the amount of I/O needed to read it.
Overview
Compression matters for databases for two reasons: it reduces how much disk (or cloud storage) a dataset consumes, and — often more importantly for analytical workloads — it reduces how many bytes have to be read from disk or over the network to answer a query. A well-compressed column can turn a multi-gigabyte scan into a fraction of that size.
General-Purpose vs. Lightweight Compression
General-purpose compressors like gzip or zstd work on arbitrary byte streams and can achieve high compression ratios, but decompressing them is relatively expensive per query. Analytical databases instead favor lightweight compression schemes that are cheap enough to decode on the fly, sometimes without fully materializing the decompressed value at all, so queries stay fast even against compressed data.
Common Techniques
- Dictionary encoding — replace repeated values with small integer codes and a separate lookup table.
- Run-length encoding (RLE) — store repeated runs as a (value, count) pair instead of repeating the value.
- Bit packing — store integers using only as many bits as their actual range requires, rather than a fixed 4 or 8 bytes.
- FSST — a string compression scheme that replaces frequent substrings with short codes, well suited to text columns.
DuckDB's Compression
DuckDB automatically chooses a lightweight compression scheme per column chunk — dictionary encoding, RLE, bit packing, constant encoding, or FSST for strings — based on which one fits that chunk's data best, with no configuration required. Because DuckDB can also read compressed Parquet files directly (typically Snappy or Zstd compressed) and push filters down into the reader, much of a query's I/O savings come before any DuckDB-specific compression is even applied.
Related terms
Run-length encoding (RLE) is a compression technique that replaces consecutive repeated values with a single (value, count) pair, shrinking runs of identical data.
Compression codecs →Compression codecs are algorithms used to reduce the size of data on disk or over the network — common examples include Snappy, Gzip, Zstandard (zstd), and LZ4 — trading off compression ratio against CPU cost and speed.
Dictionary encoding →Dictionary encoding is a compression technique that replaces repeated column values with small integer codes, storing the distinct values once in a separate lookup table.
Columnar storage →Columnar storage is a data layout that groups values from the same column together on disk, rather than storing complete rows contiguously, which speeds up analytical scans and compression.
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
FAQS
No. DuckDB automatically selects the best lightweight compression scheme per column chunk (dictionary, RLE, bit packing, FSST, or constant encoding) based on the data, with no manual tuning needed.
