Run-length encoding
Run-length encoding (RLE) is a compression technique that replaces consecutive repeated values with a single (value, count) pair, shrinking runs of identical data.
Overview
Run-length encoding takes advantage of runs — consecutive repeated values — in a dataset. Instead of storing each occurrence of a repeated value individually, RLE stores the value once along with how many times it repeats in a row. It's one of the oldest and simplest compression techniques, and it remains highly effective for columnar data that's sorted or naturally clustered.
How RLE Works
A column like ['CA', 'CA', 'CA', 'CA', 'NY', 'NY', 'TX'] becomes the pair sequence [('CA', 4), ('NY', 2), ('TX', 1)]. The longer the runs, the greater the compression ratio; a column with no repeated adjacent values compresses poorly (or not at all) under RLE.
Best-Case Data
RLE performs best on columns that are sorted or clustered by that value — a date column in a table stored in date order, or a status column where a table has been physically sorted or partitioned by status. It performs poorly on columns with high cardinality or no locality, such as a randomly ordered UUID column.
DuckDB's Use of RLE
DuckDB automatically applies run-length encoding as one of its per-column-chunk lightweight compression schemes whenever it detects that a chunk has long runs of repeated values, alongside dictionary encoding, bit packing, and FSST — chosen automatically with no configuration. Because RLE benefits so much from data locality, sorting or clustering a table on a low-cardinality column before loading it into DuckDB (or writing it out with ORDER BY in a COPY ... TO 'file.parquet') can meaningfully improve both compression ratio and scan speed.
Related terms
Data compression encodes data using fewer bits than its original representation, reducing storage footprint and the amount of I/O needed to read it.
Dictionary encoding →Dictionary encoding is a compression technique that replaces repeated column values with small integer codes, storing the distinct values once in a separate lookup table.
Compression codecs →Compression codecs are algorithms used to reduce the size of data on disk or over the network — common examples include Snappy, Gzip, Zstandard (zstd), and LZ4 — trading off compression ratio against CPU cost and speed.
Columnar storage →Columnar storage is a data layout that groups values from the same column together on disk, rather than storing complete rows contiguously, which speeds up analytical scans and compression.
Cardinality →Cardinality refers to the number of distinct values in a column or dataset, or, in data modeling, the nature of a relationship between two tables (such as one-to-many).
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
FAQS
RLE compresses consecutive repeated values into a single (value, count) pair. Sorting or clustering data on a column creates longer runs of identical values, which increases the compression ratio.
