Feather format
Feather is a fast, lightweight binary file format for storing columnar data frames on disk, based on the Apache Arrow columnar memory format, commonly used for quick interchange between pandas, R, and other Arrow-compatible tools.
Overview
Feather is a file format for saving data frames to disk with minimal serialization overhead, so that reading and writing is close to as fast as the data's in-memory representation. The current version, Feather V2, is simply the Apache Arrow IPC (interprocess communication) file format — Feather is essentially a friendly name for writing an Arrow table straight to disk.
Why it's fast
Because Feather stores data in the same columnar layout Arrow uses in memory, reading a Feather file mostly involves memory-mapping bytes rather than parsing and reconstructing values, as text formats or row-oriented formats require. This makes it well suited to short-lived, intermediate files passed between processing steps or between languages like Python and R.
Feather vs. Parquet
Feather trades some capabilities for speed. It generally applies lighter compression than Parquet and lacks Parquet's rich per-column statistics and predicate pushdown, which matter for scanning large datasets selectively. In practice, Feather is best for fast local interchange and caching, while Parquet is the better choice for long-term storage, large analytical datasets, and cross-tool archival, since it's more widely supported and compresses more aggressively.
DuckDB and Feather/Arrow files
DuckDB reads Arrow IPC files — the format Feather V2 uses — through its Arrow support, recognizing .arrow and .arrows files directly:
Copy code
INSTALL arrow;
LOAD arrow;
SELECT * FROM read_arrow('data.arrow');
In Python, DuckDB can also query a Feather file zero-copy after loading it with PyArrow, since DuckDB's Python client integrates directly with Arrow tables:
Copy code
import pyarrow.feather as feather
import duckdb
table = feather.read_table("data.feather")
duckdb.sql("SELECT * FROM table").show()
Related terms
Apache Arrow is an open-source, cross-language development platform for in-memory columnar data.
Arrow Flight →Arrow Flight is a high-performance, gRPC-based protocol from the Apache Arrow project for transferring large columnar datasets between systems with minimal serialization overhead.
PyArrow →PyArrow is a Python library that provides a high-performance interface for working with columnar data structures, particularly those defined by the Apache…
Parquet →Apache Parquet is a columnar storage file format designed for efficient data processing and analytics.
ORC →ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
FAQS
Feather is used for fast, low-overhead storage of data frames on disk, typically for short-lived intermediate files or passing data between tools like pandas and R without the overhead of a text format.
Feather V2 is the same on-disk layout as the Apache Arrow IPC file format — Feather is effectively a name for writing an Arrow table directly to a file.
Use Feather for fast, temporary local interchange between processes or languages. Use Parquet for long-term storage and large analytical datasets, since it compresses more heavily and has broader tooling and query-engine support.
