Schema registry
A schema registry is a centralized service that stores, versions, and validates data schemas — typically for Avro, Protobuf, or JSON Schema — used to keep producers and consumers in streaming systems like Kafka compatible as schemas change.
Overview
In a streaming pipeline, producers and consumers of messages are usually separate services deployed and evolved independently. A schema registry gives them a shared, versioned source of truth for what a message's structure looks like, so a producer can safely add a field or change a type without silently breaking every consumer downstream.
How it works
When a producer writes a message, it registers (or references) its schema with the registry, which returns a schema ID. That ID is embedded in the message instead of the full schema, keeping messages small. Consumers look up the schema by ID when they need to deserialize a message. The registry enforces compatibility rules — commonly backward compatibility (new schema can read data written with the old one) or forward compatibility (old schema can read data written with the new one) — and rejects schema changes that would break those guarantees.
Common implementations
The Confluent Schema Registry is the most widely used implementation, built for and tightly integrated with Apache Kafka, and it supports Avro, Protobuf, and JSON Schema. AWS Glue Schema Registry provides similar functionality for AWS-native streaming pipelines.
Where DuckDB fits
DuckDB is a query engine, not a streaming system, so it doesn't interact with a schema registry directly. In practice, schema-registry-governed messages (Avro or Protobuf records flowing through Kafka) are typically consumed and landed into files — often Parquet, via a stream processor or connector — before being queried analytically. Once that data lands as Parquet or Avro files, DuckDB can read it directly:
Copy code
INSTALL avro;
LOAD avro;
SELECT * FROM read_avro('s3://bucket/kafka-topic-exports/*.avro');
The schema evolution guarantees enforced upstream by the registry are what make it safe for that landed data to change shape over time without breaking downstream queries.
Related terms
Apache Avro is a row-based binary data serialization format that stores a data schema alongside the data, widely used for message serialization in streaming systems like Kafka and for schema evolution in Hadoop-era pipelines.
Protobuf →Protocol Buffers (Protobuf) is Google's language-neutral, binary serialization format that defines message structures in .proto schema files, compiled into strongly typed code for many programming languages, widely used in gRPC APIs and event streaming.
Schema evolution →Schema evolution is the ability to change a table's structure — adding, removing, renaming, or reordering columns, or widening types — over time without needing to rewrite all existing data or break existing readers.
schema →A schema is a logical container or blueprint that defines how data is organized within a database.
Data contract →A data contract is a formal, agreed-upon specification between a data producer and its consumers that defines the schema, format, semantics, and guarantees (like update frequency or backward-compatibility rules) of a dataset.
Apache Kafka →Apache Kafka is an open-source distributed event streaming platform built around a durable, partitioned log for publish-subscribe messaging at scale.
FAQS
A schema registry centrally stores and versions message schemas — typically Avro, Protobuf, or JSON Schema — so producers and consumers in a streaming system like Kafka stay compatible as schemas evolve over time.
The Confluent Schema Registry is the most widely used, built specifically for Apache Kafka and supporting Avro, Protobuf, and JSON Schema, with configurable backward or forward compatibility rules.
Not directly. DuckDB queries files and tables, not live Kafka streams. Data governed by a schema registry is typically landed as Avro or Parquet files by a separate stream processor, after which DuckDB can query it directly with functions like read_avro() or read_parquet().
