ORC
ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
Overview
ORC (Optimized Row Columnar) is a columnar storage format developed by Hortonworks and Facebook to improve on the limitations of earlier Hive file formats. Like Parquet, it stores data column by column rather than row by row, which improves compression and lets query engines read only the columns a query needs.
Structure
An ORC file is divided into "stripes," each holding a group of rows. Within a stripe, data is organized by column, along with lightweight indexes and statistics (min/max values, row counts) for each column. Query engines use these statistics to skip entire stripes that can't match a filter, similar to Parquet's row-group statistics.
ORC vs. Parquet
ORC and Parquet solve the same basic problem and are often interchangeable in practice. ORC has historically been closely tied to the Hive and Trino/Presto ecosystems, with some Hive-specific optimizations (like built-in support for Hive's ACID transaction format), while Parquet has broader adoption across Spark, pandas, Arrow-based tools, and cloud analytics engines. Neither format is a table format on its own — both are typically paired with a table format like Apache Iceberg or Delta Lake for transactions and schema management at the table level.
Working with ORC and DuckDB
DuckDB does not have a native ORC reader — Parquet is its primary supported columnar file format. To bring ORC data into DuckDB, a common approach is to use Apache Arrow (which does support ORC) to read the file into an Arrow table or Parquet file, and then query it from DuckDB:
Copy code
import pyarrow.orc as orc
import duckdb
table = orc.read_table("data.orc")
duckdb.sql("SELECT * FROM table LIMIT 10").show()
Alternatively, converting ORC files to Parquet with Spark or Trino ahead of time makes them directly queryable with read_parquet() in DuckDB.
Related terms
Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Apache Hive →Apache Hive is a data warehouse system built on Hadoop that lets users query large datasets stored in HDFS or object storage using HiveQL, a SQL-like language, which Hive translates into distributed jobs.
storage →DuckDB uses an efficient columnar storage format optimized for analytical queries.
Predicate pushdown →Predicate pushdown is a query optimization that moves filter conditions as close as possible to the data source, so fewer rows (or bytes) are read before filtering happens.
FAQS
ORC is a columnar file format commonly used in the Hadoop, Hive, and Trino ecosystems for storing large analytical datasets efficiently, with strong compression and predicate pushdown via per-stripe statistics.
The two formats are similar in design and performance for most workloads. ORC has deeper roots in Hive-specific features, while Parquet has wider tooling support across Spark, Arrow, pandas, and cloud query engines, which is why Parquet is more commonly seen outside the Hadoop ecosystem.
Not natively. DuckDB's primary columnar format is Parquet. To query ORC data in DuckDB, convert it to Parquet first, or load it through Apache Arrow, which does have native ORC support, and query the resulting Arrow table.
