Arrow Flight
Arrow Flight is a high-performance, gRPC-based protocol from the Apache Arrow project for transferring large columnar datasets between systems with minimal serialization overhead.
Overview
Arrow Flight is a network protocol, built on gRPC, for moving Arrow-formatted columnar data between client and server processes efficiently. Traditional database wire protocols like JDBC and ODBC serialize data row by row, which imposes significant overhead when moving large analytical result sets. Arrow Flight instead streams data already in the Arrow columnar in-memory format, so a server can send data with minimal conversion and a client can consume it directly without a costly deserialization step, enabling much higher throughput for bulk data transfer.
Flight SQL
Arrow Flight SQL is a schema built on top of Flight specifically for interacting with SQL databases — running queries, fetching metadata, and streaming results — giving database vendors a common, high-performance wire protocol that multiple clients can speak.
Relationship to ADBC and DuckDB
Arrow Database Connectivity (ADBC) is a related but distinct Arrow-project API: it's a client interface (similar in spirit to JDBC/ODBC) for talking to databases in an Arrow-native way, and it complements Flight SQL rather than replacing it — Flight SQL is a wire protocol a server can implement, while ADBC is the client-side API that can talk to that wire protocol among other transports. DuckDB has official ADBC driver support (available for Python, C/C++, Go, R, and others), which gives it a high-throughput, columnar ingestion and query path. DuckDB itself is not primarily a Flight server, but its ADBC driver and Arrow-native Python/C++ APIs mean it interoperates cleanly with tools and services in the broader Arrow Flight/ADBC ecosystem, such as ingesting Arrow record batches with minimal copying or exporting query results as Arrow tables for a downstream Flight-based service to consume.
Why it matters
For systems that need to move large volumes of analytical data between processes — a data warehouse and a compute cluster, or a database and a BI tool — Flight's zero-copy, streaming design can be dramatically faster than row-oriented protocols, which is why it has become a common building block in modern data infrastructure.
Related terms
Apache Arrow is an open-source, cross-language development platform for in-memory columnar data.
Feather format →Feather is a fast, lightweight binary file format for storing columnar data frames on disk, based on the Apache Arrow columnar memory format, commonly used for quick interchange between pandas, R, and other Arrow-compatible tools.
PyArrow →PyArrow is a Python library that provides a high-performance interface for working with columnar data structures, particularly those defined by the Apache…
connectorx →connectorx is a Rust-based Python library for loading data from SQL databases into DataFrames (pandas, Arrow, Polars) as fast as possible by parallelizing extraction and avoiding unnecessary data copies.
C/C++ →DuckDB provides a C/C++ API, allowing developers to embed the database directly into C and C++ applications.
ORC →ORC (Optimized Row Columnar) is a columnar file format originally built for Hadoop and Hive, designed for fast reads, high compression, and efficient predicate pushdown on large analytical datasets.
FAQS
No. Flight SQL is a network wire protocol a database server implements; ADBC is a client API standard (like JDBC/ODBC) that can use Flight SQL, among other mechanisms, to talk to a database in an Arrow-native way.
DuckDB's primary Arrow-ecosystem integration is through its ADBC driver and native Arrow import/export in its client APIs, rather than DuckDB itself acting as a Flight server.
