Columnar storage
Columnar storage is a data layout that groups values from the same column together on disk, rather than storing complete rows contiguously, which speeds up analytical scans and compression.
Overview
In a columnar layout, all the values for a single column are stored together, rather than each row's fields being stored contiguously as in a traditional row-oriented database. This layout matches how analytical queries actually access data: a query that aggregates revenue grouped by region only needs those two columns, and a columnar engine can read just those columns from disk while skipping every other column in the table entirely.
Columnar vs. Row-Oriented
Row-oriented storage is efficient when a workload reads or writes whole records at a time — the classic OLTP pattern of fetching one customer's full record. Columnar storage is efficient when a workload scans and aggregates over large numbers of rows but touches only a subset of columns — the classic OLAP pattern. Columnar layouts also compress much better, since values within a single column tend to be far more similar to each other (repeated categories, sorted timestamps, small integer ranges) than values across a whole row.
DuckDB's Columnar Engine
DuckDB stores tables in row groups (chunks of roughly 122,880 rows by default), and within each row group, data for each column is stored separately as a column chunk. Each column chunk is compressed independently using whichever lightweight compression scheme (dictionary encoding, run-length encoding, bit packing, or FSST for strings) fits it best, and DuckDB automatically maintains a min-max zone map per column chunk to skip chunks that can't match a filter. This columnar layout, combined with DuckDB's vectorized execution engine, is what lets it scan and aggregate large Parquet-scale datasets efficiently on a single machine.
Related terms
Row-oriented storage is a data layout that stores each record's fields contiguously on disk, so a whole row can be read or written in a single operation.
storage →DuckDB uses an efficient columnar storage format optimized for analytical queries.
OLAP vs OLTP →OLAP (Online Analytical Processing) and OLTP (Online Transaction Processing) are two workload categories: OLTP handles many small, concurrent transactional writes on normalized schemas, while OLAP handles large read-heavy analytical queries on denormalized, columnar data.
Dictionary encoding →Dictionary encoding is a compression technique that replaces repeated column values with small integer codes, storing the distinct values once in a separate lookup table.
row group →A row group is a fundamental storage concept in columnar databases like DuckDB that represents a horizontal partition of data containing a fixed number of…
Data compression →Data compression encodes data using fewer bits than its original representation, reducing storage footprint and the amount of I/O needed to read it.
FAQS
Analytical queries typically touch a handful of columns across many rows. Columnar storage lets the engine read only the needed columns from disk and compress each column efficiently, instead of reading whole rows just to use a few fields.
Yes. DuckDB stores and processes data column-by-column within row groups, which is central to its performance on analytical (OLAP) workloads.
