Schema Registry

Acts as the single source of truth for all data schemas across an organization. Every schema is stored with a unique global identifier and a monotonically increasing version number. This allows producers and consumers to negotiate data contracts precisely.

• Stores historical versions for auditability and rollback
• Enables schema fingerprinting to deduplicate identical structures
• Provides a RESTful API for registration, retrieval, and management
• Example: A Customer schema evolves from v1 (name, email) to v2 (name, email, phone) without breaking downstream services

Validates every new schema version against a configurable compatibility mode before accepting it. This is the critical gate that prevents producers from publishing data that consumers cannot deserialize, eliminating runtime failures.

• BACKWARD: New schema can read data written by previous version (default)
• FORWARD: Previous schema can read data written by new version
• FULL: Both backward and forward compatibility required
• NONE: Compatibility checks disabled (development only)
• Example: Deleting a required field violates BACKWARD compatibility and is rejected

Defines the permissible atomic operations for evolving schemas without breaking compatibility. Understanding these rules is essential for safe data contract evolution in continuous deployment environments.

• Safe Additions: Adding optional fields with default values
• Safe Deletions: Removing fields that had default values
• Breaking Changes: Renaming fields, changing data types, removing required fields
• Union Handling: Adding types to a union is backward-compatible; removing types is not
• Example: Adding a middle_name field with a default of null is a safe evolution in Avro

While deeply associated with Apache Avro, a production-grade registry supports multiple serialization formats. This allows different teams to use the optimal format for their use case while maintaining a unified governance layer.

• Apache Avro: Compact binary format with schema embedded in data files
• Protocol Buffers (Protobuf): Language-neutral, platform-neutral extensible mechanism
• JSON Schema: Human-readable schema for REST APIs and lightweight streaming
• Each format has its own compatibility rules and evolution constraints
• Example: A data engineering team uses Avro for Kafka topics while API teams register JSON Schemas

To avoid a bottleneck, schema registry clients cache schemas locally. The wire format carries only a schema ID, not the full schema definition. This decouples governance from throughput.

• Producers fetch or register a schema once and cache the returned ID
• Messages contain only the 4-byte schema ID prefix, not the schema itself
• Consumers look up the schema by ID from their local cache first
• Cache misses trigger a fetch from the central registry
• Example: A Kafka message is 1,000 bytes of Avro data prefixed by a 4-byte magic byte and schema ID, not a 5KB schema definition

Compatibility rules and normalization settings are applied at the subject level, not globally. A subject is typically a <topic>-key or <topic>-value namespace, allowing different data streams to have different governance strictness.

• High-risk financial topics enforce FULL compatibility
• Experimental data science topics use NONE during prototyping
• Configuration inheritance from global defaults with per-subject overrides
• Supports aliasing to share schemas across multiple subjects
• Example: transactions-value subject enforces BACKWARD compatibility while clicks-value allows FORWARD