Schema Registry

Manages the lifecycle of a schema as an append-only, immutable sequence of versions. Each change to a schema—adding a field, removing a field, or changing a data type—creates a new, uniquely identified version.

• Unique ID Assignment: Each schema version receives a globally unique identifier, typically an incrementing integer or a fingerprint of the schema structure.
• Immutable History: Once registered, a schema version is never deleted or mutated, ensuring that any historical message can be deserialized forever.
• Subject Strategies: Defines how schemas are grouped under a 'subject' (e.g., TopicNameStrategy, RecordNameStrategy), which determines the scope of compatibility checks.

Acts as a gatekeeper that validates every new schema version against a defined compatibility policy before accepting it. This prevents producers from writing poison pill messages that consumers cannot read.

• BACKWARD: A new schema can read data written by the previous schema. Consumers can upgrade first.
• FORWARD: The previous schema can read data written by the new schema. Producers can upgrade first.
• FULL: Combines both backward and forward compatibility. Old and new schemas are mutually readable.
• NONE: Disables all compatibility checks, allowing any change.

Provides first-class support for the three dominant serialization formats in streaming systems, each with distinct trade-offs.

• Apache Avro: A compact, binary format that carries its own schema in the file header. It is the default for many Hadoop and Kafka ecosystems due to its strong support for schema evolution.
• Protocol Buffers (Protobuf): Google's language-neutral, platform-neutral mechanism for serializing structured data. It relies on code generation from .proto definitions and is favored for gRPC services.
• JSON Schema: A vocabulary for annotating and validating JSON documents. It is used when human readability is prioritized over wire efficiency.

Plugs directly into client libraries to decouple raw bytes from structured objects. Producers use a Serializer to convert an object into bytes prefixed with a Schema ID; consumers use a Deserializer to fetch the schema from the registry and reconstruct the object.

• Schema ID Prefix: Messages in Kafka carry a 4-byte or 5-byte header containing the globally unique Schema ID, not the full schema text.
• Client-Side Caching: Deserializers cache schemas locally to avoid a round-trip to the registry for every single message, minimizing performance overhead.
• Pluggable Architecture: Custom SerDes implementations can be registered for proprietary or legacy data formats.

Allows a schema to reference other schemas, enabling modularity and reuse across complex data models. Instead of duplicating a CustomerAddress definition in every schema, it is defined once and referenced.

• Dependency Management: The registry tracks the dependency graph, ensuring that all referenced schemas are registered and compatible before a parent schema is accepted.
• Contextual Validation: When a message is deserialized, the registry resolves the entire reference tree to validate the complete object structure.
• Reduced Payload Size: By referencing common definitions, the overall schema management overhead is reduced, and consistency is enforced across the organization.

Exposes a comprehensive HTTP API for programmatic management of schemas, subjects, and configurations. This is the integration point for CI/CD pipelines and governance tools.

• CRUD Operations: Register new schemas, retrieve specific versions, delete soft-deprecated schemas, and update compatibility modes via REST endpoints.
• Role-Based Access Control (RBAC): Integrates with security frameworks to restrict who can register new schemas or change compatibility modes, preventing unauthorized schema mutations.
• Mode Management: Allows administrators to set a subject to READONLY or IMPORT mode for migration scenarios, preventing producers from registering conflicting schemas during maintenance windows.

A row-oriented remote procedure call and data serialization framework developed within the Apache Hadoop ecosystem. It is the most common serialization format paired with a schema registry due to its compact binary format and dynamic schema resolution.

• Stores the writer's schema in the file header or message, enabling the reader to resolve differences without an external lookup.
• Supports schema evolution through default values and union types.
• Tightly integrated with Apache Kafka via the Confluent Schema Registry.

Google's language-neutral, platform-neutral extensible mechanism for serializing structured data. Unlike Avro, Protobuf relies on code generation from .proto files to produce language-specific data access classes.

• Uses a binary wire format that is smaller and faster to parse than JSON.
• Schema evolution relies on field numbering and reserved fields to prevent reuse.
• Increasingly supported by schema registries to manage .proto file versions and enforce backward compatibility across gRPC services.

A vocabulary for annotating and validating JSON documents. While less compact than binary formats, it is the dominant schema language for REST APIs and webhook integrations.

• Defines expected types, required fields, and value constraints using a declarative JSON document.
• Schema registries can store JSON Schemas to validate HTTP request/response payloads and detect breaking changes like removing a required field.
• Essential for OpenAPI 3.1 specifications, which fully embrace JSON Schema for API definition.