Schema Registry

The registry acts as a gatekeeper, validating that new schema versions are backward, forward, or fully compatible with existing consumers. It prevents breaking changes like deleting a required field or altering a data type.

• Backward Compatibility: New schema can read data written by old schema.
• Forward Compatibility: Old schema can read data written by new schema.
• Full Compatibility: Both backward and forward compatible.
• Rejects incompatible schemas at registration time, preventing runtime failures.

Stores all message schemas (e.g., Apache Avro, JSON Schema, Protobuf) in a single, durable repository. Each schema is assigned a unique schema ID and version number, creating an immutable history.

• Eliminates schema copy-paste across microservices.
• Provides a single source of truth for all data contracts.
• Enables easy lookup of any schema version via REST API.
• Supports schema tagging for lifecycle management (e.g., dev, prod).

Producers embed a compact schema ID in the message payload instead of the full schema. Consumers use this ID to fetch the schema from the registry and deserialize the message.

• Reduces message size significantly compared to self-describing formats.
• Decouples data structure from the payload, enabling efficient wire protocols.
• Supports multiple wire formats like Avro binary, JSON, and Protobuf.
• Caches schemas locally on clients to minimize registry lookups.

Organizes schemas under subjects, which are logical namespaces typically mapping to a topic or message channel. This enables fine-grained access control and multi-team governance.

• A subject like fleet.telemetry.v1 isolates schemas for a specific data stream.
• Supports RBAC to control who can register, read, or delete schemas.
• Enables multiple independent teams to manage their own data contracts.
• Prevents naming collisions across different business domains.

In a heterogeneous fleet, a Schema Registry ensures that a Unified Control API command from the orchestrator is correctly interpreted by every Agent Driver, regardless of the robot's native protocol.

• Validates that MoveTo commands adhere to the canonical schema.
• Ensures telemetry from a new AMR model is compatible with the Fleet State Estimation pipeline.
• Prevents a Protocol Adapter update from breaking downstream consumers.
• Acts as the contract enforcement layer for the Message Bus.

Exposes a comprehensive REST API for all operations, enabling seamless integration with CI/CD pipelines and runtime clients. Common operations include registering a new schema, retrieving a schema by ID, and checking compatibility.

• POST /subjects/{subject}/versions: Register a new schema version.
• GET /schemas/ids/{id}: Fetch a schema by its globally unique ID.
• POST /compatibility/subjects/{subject}/versions/latest: Test compatibility.
• Native client libraries for Java, Python, Go, and C++.

The process of modifying a data schema over time while maintaining backward and forward compatibility with existing producers and consumers. A registry enforces compatibility checks—such as FULL, FORWARD, or BACKWARD—before a new schema version is committed. For example, adding an optional field with a default value is a backward-compatible change, whereas renaming a required field is a breaking change that the registry will reject.

The binary or text encoding used to structure data for transmission or storage. Common formats managed by a registry include:

• Apache Avro: Compact binary format with schema embedded in each file or via a registry ID.
• Protocol Buffers (protobuf): Language-neutral, platform-neutral mechanism from Google.
• JSON Schema: Vocabulary for annotating and validating JSON documents. The registry stores the canonical version of these schemas, decoupling the data structure from the application code.

A convention for mapping a Kafka topic or message channel to a schema subject name within the registry. Strategies include:

• TopicNameStrategy: One schema per topic, grouping all message types under a single subject.
• RecordNameStrategy: One schema per fully-qualified record type, allowing multiple event types in one topic.
• TopicRecordNameStrategy: Combines topic and record name for subject naming. This strategy is critical for organizing schemas and preventing naming collisions in a multi-service fleet.

A policy enforced by the registry at schema registration time to prevent breaking changes. A new schema version is compared against the last committed version. If a producer attempts to register a schema that violates the configured compatibility level—such as deleting a field under BACKWARD compatibility—the registry rejects the registration with an error code, protecting downstream consumers from deserialization failures.

The mechanism by which a producer prepends a 4-byte magic byte and a globally unique schema ID to each serialized message payload. Consumers extract this ID, fetch the corresponding schema from the registry's cache, and deserialize the message. This decoupling means the full schema text is never sent over the wire, reducing message size and enabling independent evolution of producers and consumers.