Schema Registry

Overview

A Schema Registry is a central service that stores the schemas for your events and checks that every change to a schema is safe. It supports formats such as Avro, Protobuf and JSON Schema. Instead of putting the full schema in every message, a producer registers the schema once, gets a small numeric id, and writes that id in front of the message. A consumer reads the id, fetches the schema from the registry, and decodes the message.

The result is a shared, versioned contract between the teams that write events and the teams that read them. The registry can refuse a new schema that would break existing readers, which turns a runtime outage into a failed deploy check. This makes schema evolution routine rather than risky. It is a core part of event-driven architecture and of Apache Kafka pipelines, and it is the natural home for the shape of any table-like data that flows through them.

What it is

A schema registry is a small service with a store and a rules engine. The store keeps schemas and their versions. The rules engine checks new versions against old ones. Clients talk to it over a simple HTTP interface and cache what they learn, so the registry is not on the hot path of every message.

The main parts

• Schema: A formal description of a record: field names, types, defaults, and sometimes documentation.
• Subject: A named scope under which schema versions are stored. With the common naming strategy, a topic called orders has a subject for its values and another for its keys.
• Version: Each new schema registered under a subject becomes the next version number.
• Schema id: A global number that identifies one schema. Messages carry this id.
• Compatibility mode: A rule that says how new versions must relate to old versions.
• Serializer and deserializer: Client libraries that talk to the registry and add or read the id in each message.

The wire format

Messages written with a registry-aware serializer usually begin with a magic byte, then a four-byte schema id, and then the encoded payload. Because the payload does not repeat the field names, messages are small. This is one reason binary formats like Avro and Protobuf are popular in event streams.

Formats

Where it came from

Early Kafka users sent JSON or custom binary strings and quickly met a painful problem. A team changed a field, and other teams' consumers broke. Nobody knew which version of a message a topic held. Avro, from the Hadoop world, offered compact binary encoding with rich schema evolution rules, and the idea of a central registry grew around it: store the schemas outside the messages, version them, and check changes.

Over time the registry idea widened. Support for Protobuf and JSON Schema was added, and the concept of a registry spread to managed cloud services as well. The core rules about backward and forward compatibility come from the older Avro specification and from general ideas in data serialisation. The registry did not invent these ideas, but it made them enforceable.

How it works

Step by step

1. A developer defines a schema for an event, for instance an OrderCreated record.
2. The producer's serializer sends the schema to the registry under a subject, such as orders-value.
3. The registry checks it against the compatibility rule for that subject. If it passes, the registry stores it as a new version and returns an id. If it matches an existing schema, it returns the existing id.
4. The serializer caches the id, then writes the magic byte, the id and the encoded payload to Kafka.
5. A consumer reads a message, takes the id, asks the registry for the schema (once, then caches it), and decodes the payload.
6. If the consumer has its own, possibly older, version of the schema, the library resolves the differences between the writer schema and the reader schema.

An everyday analogy: think of a shared dictionary kept in a library. Every book (message) just says "written in dictionary edition 47". Readers look up edition 47 once and keep a copy. The librarian (the registry) will not accept a new edition that would make old books unreadable. Nobody has to print the whole dictionary in every book.

Compatibility modes

Compatibility is the heart of the registry. Here are the common modes, with a plain description of each.

Backward compatibility is the usual default. It lets you rewind a consumer group to the start of a topic and read old messages with the newest schema. That is important for replays and for adding new consumers.

What changes are safe

For Avro, in a typical backward-compatible setting:

• Adding a field with a default value is safe.
• Removing a field that has a default is safe.
• Adding a field without a default is not safe for backward compatibility, because the new reader cannot fill it from old data.
• Changing a field type is usually not safe, except for allowed promotions such as int to long.
• Renaming a field is not safe unless you use aliases.

For Protobuf, field numbers are what matter. Never reuse a number for a different meaning, and mark removed numbers as reserved. For JSON Schema, rules depend on whether additional properties are allowed, so closed and open content models behave differently.

Subject naming strategies

Choose carefully. Changing the strategy later means new subjects and a migration.

Using it from code

Here is a Python example using a generic client. Hosts and names are placeholders.

from confluent_kafka import Producer
from confluent_kafka.schema_registry import SchemaRegistryClient
from confluent_kafka.schema_registry.avro import AvroSerializer
from confluent_kafka.serialization import SerializationContext, MessageField

schema_str = """
{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "example.orders",
  "fields": [
    {"name": "order_id", "type": "long"},
    {"name": "amount", "type": "double"},
    {"name": "currency", "type": "string", "default": "EUR"}
  ]
}
"""

client = SchemaRegistryClient({"url": "http://registry.example.internal:8081"})
serializer = AvroSerializer(client, schema_str)

producer = Producer({"bootstrap.servers": "broker.example.internal:9092"})
value = {"order_id": 1, "amount": 59.9, "currency": "EUR"}
payload = serializer(value, SerializationContext("orders", MessageField.VALUE))
producer.produce(topic="orders", value=payload)
producer.flush()

Using the REST interface

You can also manage schemas with plain HTTP calls.

# Register a schema under a subject
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"string\"}"}' \
  http://registry.example.internal:8081/subjects/demo-value/versions

# Test compatibility before registering
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"string\"}"}' \
  http://registry.example.internal:8081/compatibility/subjects/demo-value/versions/latest

# Set the compatibility mode for one subject
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD"}' \
  http://registry.example.internal:8081/config/demo-value

Where the registry stores its data

A self-run registry commonly keeps its data in a compacted Kafka topic, so the log of schemas is durable and replicated like any other topic. Managed services may use their own storage. In either case, treat the registry as critical: if it is unavailable, new schemas cannot be registered, and clients without cached ids cannot decode messages.

Key concepts and terms

A worked example

Imagine a topic orders with this first schema (version 1):

{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "example.orders",
  "fields": [
    {"name": "order_id", "type": "long"},
    {"name": "amount", "type": "double"}
  ]
}

Change 1: add an optional field

The business wants a currency. The team proposes version 2:

{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "example.orders",
  "fields": [
    {"name": "order_id", "type": "long"},
    {"name": "amount", "type": "double"},
    {"name": "currency", "type": "string", "default": "EUR"}
  ]
}

Check against BACKWARD compatibility. A consumer using version 2 reads an old message that has no currency. The default fills it in as "EUR". The check passes. The registry assigns version 2.

Change 2: add a field with no default

Another proposal, version 3, adds customer_id as a required long with no default. A version 3 reader that encounters an old message cannot fill customer_id. The registry rejects it with a compatibility error. The fix is to make the field optional with a union and a null default:

{"name": "customer_id", "type": ["null", "long"], "default": null}

Now the check passes.

Change 3: rename a field

Someone wants amount renamed to total. A plain rename fails, because the old field disappears and the new one has no default. Two safer routes exist. One is an alias on the new field, so the reader can map the old name. The other is to add total with a default, keep amount for a while, migrate producers, and remove amount later. The second route is slower but easier to reason about, and it works in every format.

Reading the numbers

Say the registry holds 3 schemas across 2 subjects. A message on orders starts with the byte 0, then the id 12, and then 14 bytes of payload. The consumer sees id 12, fetches the schema once, caches it, and decodes every later message with id 12 from memory. The registry saw one request, not millions.

A small compatibility checker in Python

This sketch shows the idea behind a backward check for Avro-like records. It is deliberately simplified.

def backward_ok(old_fields, new_fields):
    """A reader using new_fields must read data written with old_fields."""
    old_names = {f["name"] for f in old_fields}
    for f in new_fields:
        if f["name"] not in old_names and "default" not in f:
            return False, f"field {f['name']} has no default"
    return True, "ok"

v1 = [{"name": "order_id"}, {"name": "amount"}]
v2 = v1 + [{"name": "currency", "default": "EUR"}]
v3 = v1 + [{"name": "customer_id"}]

print(backward_ok(v1, v2))  # (True, 'ok')
print(backward_ok(v1, v3))  # (False, 'field customer_id has no default')

Real checkers also compare types, unions, enums and nested records. Use the registry's own check for the final answer.

A CI step

A good habit is to test every schema change in the build pipeline, before a producer ever deploys. The step reads the proposed schema file and calls the compatibility endpoint. If the answer is false, the build fails. This catches the mistake in a pull request, where it is cheap to fix.

steps:
  - name: check-schema-compat
    run: |
      python tools/check_compat.py \
        --registry http://registry.example.internal:8081 \
        --subject orders-value \
        --schema schemas/order_created.avsc

Schema references and shared types

Large organisations share common types such as an address, a money amount or an event header. Instead of copying these into every schema, a schema can point to another registered schema by reference. The registry stores the reference with a subject name and version, and resolves it when the schema is used.

Benefits of references:

• One definition of a shared type, so a fix is made once.
• Smaller schema files that are easier to review.
• Clear dependency tracking: you can see which schemas use a given type.

Costs of references:

• A change to a shared type affects every schema that uses it, so those changes need extra care.
• Tools must support resolving references, and some older clients do not.

Deleting and deprecating

Schemas can be soft deleted, which hides a version but keeps the id valid, or permanently deleted, which removes it. Permanent deletion is risky. If any message in a topic still carries that id, readers will fail. A safer approach is to mark a subject as deprecated in its documentation, stop producing to it, and wait until the topic's retention has passed before any deletion.

Security and access

Registries often sit behind authentication. Common controls include:

• Read access for all consumers, write access only for CI and a small group of owners.
• Separate subjects per team, with permissions by subject prefix.
• Encrypted connections between clients and the registry.
• Audit logs of who registered what and when.

Schemas can expose the names of sensitive fields, so treat read access with some care as well.

A small ownership table

Strengths and limitations

Strengths

• Safe evolution. Compatibility checks stop breaking changes before they reach production.
• Smaller messages. The schema is not repeated with each message.
• One source of truth. Teams look in one place for the shape of a topic.
• Better tooling. Code generation, documentation and validation all start from the same schema.
• Supports replay. With backward compatibility, new consumers can read the full history.
• Clear ownership. A subject has an owner who is responsible for changes.

Limitations

• A new dependency. If the registry is down and caches are cold, clients may fail.
• Compatibility rules do not capture meaning. Changing the unit of a field from dollars to cents passes every check but breaks every reader.
• Rules for each format differ, and teams often learn the subtleties the hard way.
• Governance takes people, not just software. Without review, the registry becomes a dumping ground.
• Schema alone does not say if values are valid. A non-null double can still be wrong or out of range, unless extra rules are added.
• Mixed topics (many record types in one topic) need the right naming strategy, and some tools do not support them well.
• Moving data between clusters needs care: ids are assigned per registry, so copying messages alone is not enough. See MirrorMaker 2 and MSK Replicator.

How it compares with alternatives

The registry and the drift tools do different jobs. A registry prevents bad changes at write time. Drift detection notices change in data that was already produced, perhaps by systems that do not use the registry. Mature teams use both.

Role in tabular foundation models

Tabular foundation models learn from raw tables. They read rows and columns, deal with mixed numeric and categorical types and missing values, may not rely on column names, and are pre-trained on large numbers of synthetic or real tables. At use time they learn from the rows you provide through in-context learning. A schema registry does not run any of this, but it controls something these models depend on: the exact shape and meaning of the columns that arrive.

Stable columns, stable rows

A table is only a table if the columns stay put. A registry keeps the event schema that feeds a table stable and versioned. When a new column appears, it comes with a default and a version. When a column is removed, the change is planned. For a model that is refreshed over time, this is the difference between a smooth update and a mysterious drop in accuracy because a column silently changed meaning.

Mixed types and logical types

Avro and Protobuf give each field a declared type, including logical types such as decimals and timestamps. This helps the step from events to tables: numeric columns land as numbers, categorical columns as strings or enums, and timestamps as timestamps. A tabular model can handle mixed types, but if types are declared up front, the model-facing table does not need guesswork. Enums in particular tell you the full set of categories, which helps encoding.

Missing values

A schema can say clearly whether a field may be null, and what the default is. That distinction matters for models. A union with null means "this value may be missing", and a default means "if missing, assume this". Mixing the two ideas leads to fake values that look real. A model that handles missing values or does imputation should see true missing values as missing. Be careful with defaults: a default of zero in a numeric field turns "not provided" into "zero", and the model cannot tell them apart. Prefer nullable fields with null defaults for measurements.

No column names

Some tabular models can work when column names are meaningless, and others use names as a clue. The registry gives names and documentation to every field. If you plan to use a name-aware model, rich names and doc strings help. If you plan to hide names, keep a separate, versioned mapping from schema field to anonymous position, so you can always trace a column back to its meaning.

Pre-training on real and synthetic tables

Pre-training corpora are assembled from many sources. Where those sources are event streams, the registry is a catalogue of what each stream means. Schemas with documentation, units and ownership make it easier to decide which tables are suitable and which carry sensitive fields. Synthetic tables can be generated to match a schema, which is a useful way to test a pipeline without real data. See /model/research for the research background, and /model/architecture for how tables are represented inside the model.

In-context learning

In-context learning relies on example rows that mean the same thing as the rows being predicted. If a producer changes a field from dollars to cents without a version bump, the example rows and the target rows disagree. Version-aware extraction, which pulls rows by schema version and normalises them, protects the context. Include the schema version as lineage metadata in each extract so you can reproduce the context later.

Continual learning

Continual learning means updating a model as new data arrives. The data changes shape over time, and the registry holds the history of those shapes. A pipeline can read old and new versions, apply the right resolution, and produce a consistent table. Backward compatibility with transitive checks is especially helpful here, since you may be reading messages written years apart.

Imputation and anomaly detection

Imputation and anomaly detection both depend on knowing which values are expected. A schema can encode ranges and enums in some formats, or in extra rules, and these become guardrails. A value outside an enum is not an anomaly of behaviour; it is a contract violation, and it should be caught at the door. Keep the two ideas apart: contract checks catch structure problems, while a tabular model catches statistical oddities.

Sector and semantic classification

When you classify columns or records into sectors or semantic types, field names, doc strings and enums supply strong hints. A well-documented schema is a free feature source. A column called naics_code with a documented meaning tells a classifier much more than col_17. Keep documentation in the schema and treat it as data.

Concrete places the registry fits in a model pipeline

• Event to table conversion: sinks read the registry to create columns of the right types in Apache Iceberg or Delta Lake tables.
• Training extracts: queries record the subject and version used, which feeds data lineage.
• Feature checks: data contract validation uses schemas as the base for further rules.
• Early warning: schema drift detection compares live data to the registered schema.
• Reshaping at the edge: single message transforms change records before the converter registers their schema.

For how models perform on public suites of tables, see /model/benchmarks. Those results assume clean, consistently typed inputs, and a registry is one of the tools that keeps inputs that way.

Practical guidance and best practices

Schema design

• Use meaningful names and add documentation strings to fields.
• Give every new field a default, or make it nullable, from the start.
• Use enums with care. In Avro, adding an enum symbol can break readers unless a default symbol is set.
• Prefer logical types for money, dates and timestamps. Avoid floating point for currency.
• Keep events small and focused. One event type, one clear meaning.
• Avoid putting large blobs in events. Send a reference instead.

Compatibility strategy

• Start with BACKWARD, or BACKWARD_TRANSITIVE if you replay full history.
• Use FULL when producers and consumers upgrade in any order.
• Do not use NONE outside of throwaway experiments.
• Set the mode per subject when needs differ, and set a global default.

Process

• Register schemas through CI, not from application start-up code in production. Turn off auto-registration for production clients.
• Review schema changes like API changes: with an owner and a description.
• Keep schemas in version control next to the code that uses them.
• Publish a short guide for naming and evolution rules.
• Monitor registry health and latency, and run it with more than one instance.

Reliability

• Cache on the client side, and test what happens when the registry is unreachable.
• Back up the schema store, or rely on the replicated log it uses.
• Plan how ids are handled during migration or mirroring between clusters, so consumers in the target region can decode messages.
• Secure the registry with authentication and authorisation. Schemas can reveal sensitive business detail.

Meaning, not just shape

• Document units and time zones in field docs.
• Add value-level rules where your tooling supports them, such as ranges, regular expressions and required fields.
• Pair the registry with data quality checks, because shape alone is not meaning.

Common mistakes

• Leaving auto-registration on in production, so any client can publish a new schema.
• Adding a required field with no default and then wondering why the registry refuses it.
• Reusing a Protobuf field number for a new meaning.
• Changing units or semantics without changing the schema, since the checker cannot see it.
• Using the wrong subject naming strategy for a topic that carries many event types.
• Assuming the registry guards data that bypasses it. Producers that write plain JSON are not checked.
• Forgetting that copying topics between clusters does not copy the meaning of schema ids.
• Treating the registry as optional infrastructure with no backups or redundancy.
• Setting NONE to get past a failed check instead of fixing the schema.
• Using defaults that look like real values, such as zero for a missing measurement.
• Making every field a union with null to avoid conflicts, which hides real requirements.

Frequently asked questions

What does a Schema Registry do?

It stores versioned schemas for your events and checks that each new version is compatible with earlier ones. Producers register a schema and receive an id, messages carry that id, and consumers use it to fetch the schema and decode the data. This keeps teams from breaking each other when events change.

Which formats does a Schema Registry support?

The common registries support Avro, Protobuf and JSON Schema. Avro is popular for its strong evolution rules and compact binary form. Protobuf is widespread in service-to-service systems, and JSON Schema is chosen when readability matters more than size.

What is the difference between backward and forward compatibility?

Backward compatibility means a new schema can read data written with the old one, so consumers can upgrade first. Forward compatibility means an old schema can read data written with the new one, so producers can upgrade first. Full compatibility requires both, which allows any upgrade order.

Why use schema ids instead of sending the schema with each message?

Sending the whole schema in every message wastes space and bandwidth. A small id is enough, because the consumer can fetch and cache the schema once. This keeps messages compact and allows central checks on every change.

What happens if the registry goes down?

Clients that have already cached the schemas they need can keep working. New schemas cannot be registered, and clients that meet an unknown id cannot decode the message until the registry is back. For this reason, run several instances, monitor them, and test client behaviour during an outage.

Can I rename a field safely?

Not with a simple rename. Avro supports aliases that let a reader map an old name to a new one, but the safest general path is to add the new field with a default, move producers and consumers over, and then remove the old field in a later version. This works in every format and keeps history readable.

Does a schema registry validate the values in my data?

Mostly it validates structure and types, not meaning. A value can have the right type and still be wrong. Some registries and companion tools allow extra rules, and data quality checks can cover ranges and relationships. Treat the registry as the first layer, not the only one.

How does a registry help machine learning on tabular data?

It keeps the columns that feed a table stable, typed and documented. That gives models consistent inputs across time, makes it easier to separate true missing values from defaults, and records which schema version produced each training extract. These properties help with reproducibility and with safe model refresh.

Do I need a registry if I use JSON?

You can use JSON Schema with a registry, and gain compatibility checks without switching to a binary format. If you use plain JSON with no schema at all, you lose enforcement, and breaking changes are found by consumers in production. For anything beyond a small prototype, a registry is worth having.

How does a registry work across regions or clusters?

Each registry assigns its own ids, so simply copying raw messages to another cluster may leave consumers there unable to decode them. Replication tools and registry features handle this in different ways, so check the behaviour of your replication path before relying on it. Plan schema migration as part of any disaster recovery design.

Glossary

• Alias: An alternate name for a field, used to support renames.
• Avro: A binary data format with rich schema evolution rules.
• Backward compatibility: New readers can read old data.
• Compatibility mode: The rule applied to new schema versions.
• Data contract: An agreed description of data, including shape, rules and ownership.
• Default value: The value used when a field is absent in the data.
• Forward compatibility: Old readers can read new data.
• Logical type: A meaning layered on a base type, like decimal or timestamp.
• Magic byte: A leading byte that marks the registry wire format.
• Protobuf: A binary format whose fields are identified by numbers.
• Reader schema: The schema a consumer uses to read data.
• Schema id: The global number carried in each message.
• Subject: The named scope for a series of schema versions.
• Transitive check: A compatibility test against every earlier version.
• Writer schema: The schema used to write the data.

Summary

A Schema Registry is a central, versioned store for event schemas, backed by compatibility rules that stop breaking changes before they ship. Messages carry a small id, schemas are fetched once and cached, and teams share one contract per topic. Avro, Protobuf and JSON Schema are all supported, each with its own rules about safe change.

For tabular foundation models, the registry is the guardian of column shape, type and meaning over time. It helps separate real missing values from defaults, keeps training and scoring extracts consistent, records lineage, and supports safe refreshes as data evolves. Use it with careful naming, CI-based registration, sensible defaults, and companion tools for drift detection and data contract validation. Related reading: Apache Kafka, Kafka Connect, Amazon MSK, and event-driven architecture.

Related

It belongs to Event streaming in my stack, under Data engineering and lakehouse. Others in the same group: