Avro
Apache Avro is a row-based binary data serialization format that stores a data schema alongside the data, widely used for message serialization in streaming systems like Kafka and for schema evolution in Hadoop-era pipelines.
Overview
Apache Avro is a binary serialization format built for compactness and schema evolution. Unlike Parquet or ORC, Avro stores data row by row rather than column by column, which makes it a good fit for write-heavy, record-at-a-time use cases like event streaming, rather than analytical scans across many rows.
Schema and evolution
Every Avro file or message is associated with a schema, defined in JSON, describing its fields and types. Avro readers and writers can use different (but compatible) schema versions — a reader can be given the writer's schema alongside its own and Avro will resolve the differences, which is what makes Avro well suited for streaming pipelines where producers and consumers evolve independently over time.
Common use cases
Avro is the default serialization format for many Apache Kafka deployments, often paired with a schema registry that governs which schema versions are compatible. It also has a long history in the Hadoop ecosystem as a row-oriented alternative to columnar formats like ORC and Parquet.
Avro vs. Parquet
Because Avro is row-based, it's efficient for appending and reading whole records — a natural fit for streaming ingestion. Parquet's columnar layout instead favors analytical queries that scan a subset of columns across many rows. A common pattern is to land streaming data as Avro and later convert it to Parquet for analytics.
DuckDB and Avro
DuckDB's avro extension adds a read_avro function that reads Avro files directly, including from remote storage:
Copy code
INSTALL avro;
LOAD avro;
SELECT * FROM read_avro('s3://bucket/events/*.avro')
WHERE event_type = 'purchase';
This lets DuckDB and MotherDuck query Avro data landed by Kafka Connect or similar ingestion tools without a separate conversion step to Parquet first.
Related terms
A schema registry is a centralized service that stores, versions, and validates data schemas — typically for Avro, Protobuf, or JSON Schema — used to keep producers and consumers in streaming systems like Kafka compatible as schemas change.
Data serialization →Data serialization is the process of converting in-memory data structures into a format that can be stored on disk or transmitted over a network, and later reconstructed through deserialization.
Schema evolution →Schema evolution is the ability to change a table's structure — adding, removing, renaming, or reordering columns, or widening types — over time without needing to rewrite all existing data or break existing readers.
Protobuf →Protocol Buffers (Protobuf) is Google's language-neutral, binary serialization format that defines message structures in .proto schema files, compiled into strongly typed code for many programming languages, widely used in gRPC APIs and event streaming.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
FAQS
Avro is commonly used to serialize messages in streaming systems like Apache Kafka and to store data in Hadoop-era pipelines, particularly where schemas change over time and row-at-a-time reads and writes are common.
Avro is a row-based format — it stores complete records one after another, which differs from columnar formats like Parquet and ORC that group values by column for analytical scans.
Yes. DuckDB's avro extension provides a read_avro() function that reads Avro files directly from local disk or remote object storage, including glob patterns for multiple files.
