IM Engineer Study Notes
← Back to list
AI & Data
#스키마 레지스트리#스키마 진화#호환성#Avro#이벤트 기반 아키텍처
Last updated · 2026-10-25

Schema Registry and Schema Evolution

1. Overview

A Schema Registry is a metadata management system that centrally registers and version-controls the data structures (schemas) of event streams and messages, verifies that messages exchanged between producers and consumers conform to an agreed schema, and enforces safe evolution according to compatibility rules whenever a schema changes.

As event-driven architectures (EDA) built on Apache Kafka have spread, it has become common for a single topic to be produced and consumed simultaneously by dozens of services. Yet a message body is nothing but a byte array, so when a producer adds a field or changes a type, an unaware consumer fails deserialization or silently reads wrong values. This phenomenon—where services deployed at different times implicitly assume the same data structure and then diverge—is called schema drift, and it is a leading cause of operational failure in distributed systems.

The traditional approach was for producers and consumers to hardcode and share schemas through a common library. But this creates two fundamental problems. First, every schema change requires redeploying all services simultaneously, breaking the loose coupling that is the essence of EDA. Second, transmitting the entire schema with each message bloats the payload. A schema registry solves both at once by letting a schema be registered once and referenced by an integer ID. Only the schema ID—about 5 bytes—rides on the message, while the actual structure definition is looked up from the registry.

The core value of a schema registry is deployment-order independence. It fixes, by prior contract, what "can change" and what "cannot change" so that message compatibility is not broken even when producers and consumers are deployed at different times, unaware of each other. This goes beyond a mere validator and is evolving into a governance tool that treats data as a contract.

In short, a schema registry has the following four characteristics.

  • Centralized registration and versioning: schemas are registered per subject and assigned versions and global IDs.
  • Compatibility enforcement: policy is checked at registration time to block breaking changes in advance.
  • Reference efficiency: only the schema ID rides on the message, minimizing payload.
  • Loose coupling: producers and consumers can be deployed independently.

This concern is essentially the same as API versioning in synchronous REST APIs. In REST, a version is stated explicitly via the URI, such as /v1 or /v2, and the client chooses the version it wants; but in an asynchronous event stream, a consumer cannot "choose" a message and must receive all past and present messages piled in the topic, in order. Therefore version management in the event world must be solved not by a "client-chooses" model but by a compatibility guarantee that "any version reads without breaking," and this is exactly the role the schema registry plays. In other words, it is accurate to understand a schema registry as an API-contract management layer for event-driven architectures.

2. Structure and Operation of a Schema Registry

A schema registry is a lookup service composed of producer, consumer, serializers/deserializers (SerDes), and a store. Before sending a message, the producer registers the schema and obtains an ID, prepends it to the message; the consumer reverse-looks-up the schema by that ID to interpret the message safely. The structure diagram below shows the relationship among all components.

graph LR
  subgraph P["Producer side"]
    PA["Application"] --> PS["Serializer(SerDes)"]
  end
  subgraph R["Schema Registry"]
    API["REST API"] --> STORE["Schema store(Kafka topic/DB)"]
    API --> CHK["Compatibility checker"]
  end
  subgraph C["Consumer side"]
    CS["Deserializer(SerDes)"] --> CA["Application"]
  end
  PS -->|"register/lookup schema"| API
  PS -->|"ID+payload"| BROKER["Message broker(Kafka)"]
  BROKER -->|"ID+payload"| CS
  CS -->|"lookup schema by ID"| API

The serializer/deserializer (SerDes) is the client library that communicates with the registry and is the most important element. On the producer side, just before sending, the serializer registers the object's schema with the registry (or confirms the ID from a cache), prepends a "magic byte + 4-byte schema ID" to the message, and binary-encodes the body. The consumer side strips off the ID and fetches the schema from the registry (caching it locally once received) to deserialize. As a result only the schema ID travels over the network, and registry lookups occur only on cache misses, so throughput is barely affected.

The schema store keeps registered schemas, versions, and subject metadata. Confluent's implementation uses an internal Kafka topic called _schemas as a log instead of a separate DB, and registry instances replay it to restore identical state. The registry thereby inherits Kafka's replication and durability directly. A subject is the namespace for a schema's versions and typically takes the form <topic>-value or <topic>-key (TopicNameStrategy). A single subject holds multiple versions such as v1, v2, v3, and each version is assigned a globally unique ID.

The compatibility checker compares a new schema against existing versions when registration is attempted and rejects it if the configured compatibility policy is violated. For example, if a producer tries to delete a field that some consumer reads as required, the registry blocks it at the registration stage, stopping the failure before deployment. This "registration-time gate" makes a schema registry not a mere store but a governance control point. In fact, real-time trading pipelines in finance include compatibility checks in the CI pipeline, failing a non-compatible schema-change PR at the build stage (so-called schema-as-code).

Looking concretely at the byte layout upon transmission reveals the efficiency. A Confluent-format message is composed of "magic byte (0x00) 1 byte + schema ID 4 bytes + actual payload." That is, instead of carrying the entire schema definition (a JSON/Avro schema that can run from hundreds of bytes to several KB) with every message, a 5-byte prefix identifies which structure it is. On a topic streaming hundreds of thousands of events per second, this difference cuts network bandwidth and storage cost by dozens of times. A consumer queries the registry only once for an ID it sees for the first time and uses the local cache thereafter, so the registry sits only on the control plane, not on the traffic path. This design is the key to achieving "centralized governance" and "distributed high throughput" at the same time.

3. Schema Evolution and Compatibility Policy

The core question of schema evolution is "when a schema changes, who must be deployed first to be safe." The compatibility policy prescribes this deployment order. The state flow below shows the compatibility-check process.

flowchart TD
  A["New schema registration request"] --> B{"Check compatibility mode"}
  B -->|"NONE"| G["Skip check and register"]
  B -->|"BACKWARD"| C["Can the new schema read old-version data?"]
  B -->|"FORWARD"| D["Can the old schema read new-version data?"]
  B -->|"FULL"| E["Both directions possible?"]
  C -->|"yes"| G
  C -->|"no"| F["Reject registration(409)"]
  D -->|"yes"| G
  D -->|"no"| F
  E -->|"yes"| G
  E -->|"no"| F
  G --> H["Assign new version number and global ID"]

Confluent Schema Registry provides seven compatibility modes (BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, NONE). The default is BACKWARD. When the _TRANSITIVE suffix is attached, the check target expands from "the single previous version" to "all versions registered so far," so data written under any past version can be handled safely.

Mode Allowed changes (representative) Check basis Upgrade order
BACKWARD (default) delete field, add field with default previous version consumers first
BACKWARD_TRANSITIVE same as above all versions consumers first
FORWARD add field, delete field with default previous version producers first
FORWARD_TRANSITIVE same as above all versions producers first
FULL add/delete field with default previous version any order
FULL_TRANSITIVE same as above all versions any order
NONE any change no check no guarantee

Backward compatibility (BACKWARD) is the rule that "the new schema must be able to read old data." Hence consumers are upgraded to the new version first, then producers. Changes that delete a field or add a field with a default are allowed. The reason Confluent makes this the default is that even when rewinding a topic to the beginning to reprocess, the latest consumer can read all past messages. It is the safest choice in practice, where log reprocessing and failure recovery are frequent.

Forward compatibility (FORWARD) is the opposite rule: "old schema must be able to read new data," and producers are upgraded first. It suits situations where an old-version consumer must read a new-version producer's messages without breaking—for example, B2B integrations where an external partner controls the consumer and cannot upgrade immediately. Full compatibility (FULL) requires both directions, freeing the deployment order itself, but narrows the allowed changes to "adding and deleting fields that have defaults." It is used to maximize safety in large organizations where control is hardest.

A breaking change that cannot pass the compatibility check is sometimes unavoidable. In that case, instead of forcing the same subject to evolve, you create a new topic (e.g., orders.v2) to run new and old streams in parallel and migrate consumers gradually—the dual-write / expand-and-contract pattern. The producer writes to both topics for a while, and after all consumers have moved to the new topic, the old one is retired. The schema registry independently guarantees each topic's compatibility even during this transition, enabling large-scale migrations without downtime.

The practical implication is clear. The choice of compatibility mode is not a technical preference but an organizational deployment-governance decision. If one team controls both producers and consumers, BACKWARD suffices; but if consumers are scattered across many organizations and cannot be controlled, you must raise it to FULL_TRANSITIVE to strongly guarantee "never breaks." Meanwhile NONE disables checks, so it is forbidden outside early development; abusing NONE in production reduces the registry to merely an "expensive version store."

4. Comparison of Serialization Formats (Avro, Protobuf, JSON Schema)

A schema registry is not tied to a particular serialization format, but each format handles schema evolution differently. Since Confluent 5.5 all three formats are supported, and the rules for "what counts as a compatible change" differ per format. For example, under FORWARD mode Protobuf forbids adding new message types—one of several format-specific constraints.

Aspect Apache Avro Protobuf JSON Schema
Encoding binary (schema separate) binary (tag numbers) text (JSON)
Evolution identity field name + default field number property name
Size/speed small and fast small and fast large and slow
Readability low low high
Best domain data pipelines/analytics gRPC/high-perf RPC web/REST-friendly

Avro transmits data separately from the schema and, at deserialization, uses both a "writer schema" and a "reader schema" to map fields by name. Thanks to this structure its schema-evolution rules are clear, and it has become the de facto standard for big-data storage (Hadoop/data lakes). However, default design is the heart of its evolution, so omitting a default on an added field breaks backward compatibility.

For example, suppose you add an optional field coupon_code to an order event in Avro. To keep backward compatibility (BACKWARD), you must specify a default for this field (e.g., "default": ""). Only then can a new-version consumer fill the default when reading a past message that lacks the field and succeed at deserialization. If you omit the default, the registry rejects registration itself (HTTP 409), surfacing the design mistake before it reaches production. Thus "default design" is the most common pitfall of Avro evolution and the core device of compatibility.

Protobuf identifies fields not by name but by integer tag numbers. Hence as long as the number is kept, the name can be changed while staying compatible, and combined with gRPC it excels at low-latency inter-service communication. However, reusing a once-used tag number silently corrupts data, so the discipline of permanently sealing a deleted number as reserved is essential. JSON Schema is text-based, so it is human-readable and friendly to the web ecosystem, but its payload is large and parsing is slow, making it unsuitable for ultra-high-throughput streaming. In practice it is common to mix by domain—"Avro for large analytics pipelines, Protobuf for internal microservice RPC, JSON Schema for externally published API events."

5. Comparison of Implementations

Representative implementations include Confluent Schema Registry (the de facto standard in the Kafka ecosystem), AWS Glue Schema Registry (managed, integrated with Kinesis/MSK), and the open-source Apicurio Registry (broadly supporting Avro, Protobuf, JSON Schema, and even OpenAPI/AsyncAPI). Confluent is deeply integrated with Kafka on a _schemas topic and has high operational maturity; Glue is serverless with little operational burden and binds naturally with IAM. Apicurio suits organizations avoiding vendor lock-in or wishing to manage API specs together.

As an example of scale, Netflix/LinkedIn-class organizations process trillions of events per day and run thousands of subjects. At this scale, without schema ID caching, registry redundancy, and an approval workflow that treats schema changes like code review, a single trivial field change can halt the enterprise-wide pipeline. In fact, one fintech case reported that after integrating compatibility checks into CI for payment-event schemas, schema-related operational incidents dropped from several per quarter to near zero (to be understood as a general trend).

Summarizing the selection criteria is as follows. The three implementations share the essence of a compatibility model but diverge in where operational responsibility lies and in supported scope.

Aspect Confluent AWS Glue Apicurio
Storage Kafka _schemas topic managed (serverless) choice of SQL DB/Kafka, etc.
Operational burden self-operated (self-hosted) almost none (fully managed) self-operated
Supported specs Avro, Protobuf, JSON Schema Avro, Protobuf, JSON above + OpenAPI, AsyncAPI
Best situation Kafka-centric, high maturity AWS-bound, simplified ops multi-spec, vendor-neutral

Subject naming strategy is also an important design decision in practice. The default TopicNameStrategy (<topic>-value) is simple when a topic carries only one kind of event, but when a single topic must mix multiple event types while preserving order, you must switch to RecordNameStrategy (by record type name) or TopicRecordNameStrategy (topic + record name). Choosing the wrong strategy makes different events share the same subject and causes compatibility checks to work incorrectly, so it must be decided early in topic design.

6. Deep Dive: Evolution into Data Contracts

Recently the schema registry has been re-evaluated as a central technology in the discussion of Data Contracts and Data Mesh. A data contract is "an explicit, version-controlled promise between the team producing data and the team consuming it," including not only schema (structure) but also semantics, quality expectations (SLO), and ownership. The schema registry handles the machine-enforceable structural part of this contract. In a data mesh, when each domain publishes data as a "data product," the compatibility policy is like the product's API stability grade.

Another sign of maturity is schema references. Common types (e.g., Address, Money) are defined as separate schemas, and multiple event schemas reference them, reusing and standardizing data definitions across the whole organization. This amounts to managing the "shared kernel" between microservices declaratively in the registry rather than in a code library, reducing semantic inconsistencies from duplicate definitions. However, as references deepen, the blast radius of a change widens, so common schemas need hierarchical governance applying a stricter compatibility policy (FULL_TRANSITIVE) and a change-approval process.

Confluent responds to this by extending toward Data Contracts features that combine metadata/tags on a schema (e.g., PII marking), rule-based validation/transformation, and migration rules. For example, marking a personal-information (PII) field with a schema tag can be wired so that downstream pipelines automatically apply masking/encryption rules. This shows the schema registry is being promoted from a mere serialization aid to an execution point for data governance and privacy. Going forward, the schema registry is expected to integrate with data-catalog/lineage systems to automatically analyze (impact analysis) the downstream effects of a schema change.

7. Considerations and Implications

  • Compatibility policy is an organizational-design problem. From a professional engineer's view, the choice among BACKWARD/FORWARD/FULL must be derived not from performance but from the organizational structure of "who controls producers and consumers." A single internal team defaults to BACKWARD; multi-organization/external integrations default to FULL_TRANSITIVE, and a design that differentiates policy per subject is desirable.
  • Manage schemas like code (schema-as-code). Schema changes should be handled with Git PRs and CI compatibility checks to block breakage before deployment. The moment NONE mode is allowed in production, the governance value of the registry disappears, so NONE should be prohibited by organizational policy and exceptions placed under an approval process.
  • Make the trade-offs of format choice explicit. Avro (analytics/pipelines), Protobuf (low-latency RPC), and JSON Schema (external public APIs) each have different evolution rules and performance traits. Format-specific pitfalls—Protobuf's prohibition on tag-number reuse, Avro's default design—must be documented as guidelines to prevent operational incidents.
  • Single point of failure (SPOF) and availability. Because the serializer depends on the registry, a registry failure can halt all production and consumption. Design redundancy, local schema caching, and a read-only fallback, and match the registry's availability SLO to the level of the message broker.
  • Governance and privacy linkage outlook. Schema-tag-based PII identification, data contracts, and lineage/catalog integration expand the schema registry into an execution point for data governance. Going forward, in a data mesh environment it will take its place as a core axis automating the quality, compatibility, and privacy control of each domain's data product.

References

In one line: A schema registry centrally version-controls the data structures of event streams and enforces safe schema evolution through compatibility policies (BACKWARD, FORWARD, FULL), loosely coupling producers and consumers and evolving into an execution point for data contracts and governance.