Data serialization
Data serialization is the process of converting in-memory data structures into a format that can be stored on disk or transmitted over a network, and later reconstructed through deserialization.
Overview
Programs represent data in memory using structures specific to their runtime — objects, structs, data frames — that can't be written to a file or sent over a network as-is. Serialization converts that in-memory representation into a defined byte or text layout; deserialization reverses the process, reconstructing usable data structures from those bytes.
Text vs. binary serialization
Text formats like JSON and XML are human-readable and easy to debug, but verbose and slower to parse. Binary formats like Avro, Protobuf, and Parquet are more compact and faster to process, at the cost of not being readable without tooling. The choice usually comes down to whether human readability, interoperability, or raw performance matters most for a given use case.
Row vs. columnar serialization
Serialization formats also differ in whether they lay out data by row or by column. Row-based formats like Avro and Protobuf serialize whole records together, which suits streaming and message-passing where a full record is produced or consumed at once. Columnar formats like Parquet serialize each column separately, which suits analytical workloads that scan and aggregate specific columns across many records.
Schema handling
Some formats embed a schema alongside the data (Avro, Parquet), letting any reader interpret the bytes without external information. Others (like Protobuf) rely on schema definitions compiled separately into code, and JSON is typically schemaless, with structure only implied by convention.
Serialization in DuckDB
DuckDB reads and writes several serialization formats natively as part of everyday SQL, without a separate library:
Copy code
-- Read JSON, a schemaless text serialization
SELECT * FROM read_json('events.json');
-- Convert to Parquet, a compact columnar binary serialization
COPY (FROM read_json('events.json')) TO 'events.parquet' (FORMAT parquet);
-- Serialize a row back to JSON text
SELECT to_json(row(id, name)) FROM users;
Related terms
Apache Avro is a row-based binary data serialization format that stores a data schema alongside the data, widely used for message serialization in streaming systems like Kafka and for schema evolution in Hadoop-era pipelines.
Protobuf →Protocol Buffers (Protobuf) is Google's language-neutral, binary serialization format that defines message structures in .proto schema files, compiled into strongly typed code for many programming languages, widely used in gRPC APIs and event streaming.
Parquet vs CSV →Parquet is a compressed, columnar binary file format optimized for analytics, while CSV is a plain-text, row-based format with no built-in schema or compression — Parquet is generally faster and smaller for analytical workloads, while CSV is simpler and universally readable.
PyArrow →PyArrow is a Python library that provides a high-performance interface for working with columnar data structures, particularly those defined by the Apache…
Schema registry →A schema registry is a centralized service that stores, versions, and validates data schemas — typically for Avro, Protobuf, or JSON Schema — used to keep producers and consumers in streaming systems like Kafka compatible as schemas change.
Table format →A table format is a metadata layer on top of data files in a lake or object storage that defines what constitutes a table — its schema, partitioning, and file list — enabling ACID transactions, schema evolution, and time travel across multiple query engines.
FAQS
Serialization converts in-memory data structures into a storable or transmittable byte format. Deserialization is the reverse process — reconstructing usable in-memory data structures from those bytes.
Text formats like JSON and XML are human-readable but larger and slower to parse. Binary formats like Avro, Protobuf, and Parquet are more compact and faster to process, but require specific tooling to read.
Yes. DuckDB can read and write several serialization formats directly through SQL, including JSON (read_json, to_json), CSV, and Parquet, without requiring a separate serialization library.
